northern greece diabetes meeting -...
TRANSCRIPT
Irreproducibility of published research in the
biosciences: From etiology to therapeutics.
Northern Greece Diabetes MeetingNov 19, 2016
Jeffrey S. Flier, M.D.
Irreproducibility: The Big Picture(basic and preclinical research)
Recent progress in biomedical sciences has been remarkable –- understanding and treatment of human disease.
Irreproducibility: The Big Picture(basic and preclinical research)
Recent progress in biomedical sciences has been remarkable –- understanding and treatment of human disease.
Growth in magnitude, scope, professionalization
› 1,000,000 publications/yearThousands of journals exist
› $100B/year in spent on biomedical research in the U.S.
Very complex ecosystem
Irreproducibility: The Big Picture(basic and preclinical research)
There will always be errors, dead ends, mistaken claims and other problems;
- the “self-correcting” nature of science is expected to deal with these.
Perfect reproducibility is both impossible and undesirable.
If we demanded perfect reproducibility, progress wouldbe reduced.
Irreproducibility: The Big Picture(basic and preclinical research)
However………
There is Increasing evidence that bioscience research published today
has a level of irreproducibility that is excessive and alarmingly high.
Irreproducibility: The Big Picture(basic and preclinical research)
However………
There is increasing evidence that bioscience research published today has a
level of irreproducibility that is excessive and alarmingly high.
-- Some (John Ioannidis) claim more than 75% of papers are false.
Irreproducibility: The Big Picture(basic and preclinical research)
However………
There is increasing evidence that bioscience research published today has a
level of irreproducibility that is excessive and alarmingly high.
-- Some claim more than 75% of papers are false.
If true, this fact would diminish progress from investment in research, and
threaten the confidence in research by funders and the public.
My Perspective on These Issues
• 40 years as physician scientist
• Role in publishing as reviewer and editor
• Academic leadership
• Dean of Harvard Medical School (9 years, recently completed term)
9,500 full time faculty – promotions and appointments
Large and robust research program
Oversaw investigations of research misconduct
Occasional spokesperson for biomedical research
Data Reproducibility: Case Study #1 : Betatrophin
Hypothesis: Severe Insulin Resistance causes beta cell hyperplasia -evidence that a circulating factor is involved. (S. Flier, Kulkarni, Kahn - 2001).
Data Reproducibility: Case Study #1 : Betatrophin
Hypothesis: Severe Insulin Resistance causes beta cell hyperplasia -evidence that a circulating factor is involved. (S. Flier, Kulkarni, Kahn - 2001).
D. Melton lab (chair of Harvard stem cell dept) reports a new hormone - “betatrophin”
- mRNA massively induced in liver of mice with severe insulin resistance;- administering the encoded protein –”betatrophin”- induces marked beta cell
hypertrophy in treated mice (Cell, 2013).
But then, one year later ……………………………
Gusarova et al , Cell 2014 – “betatrophin does not control beta cell expansion” ;
- repeated experiment with no beta cell response; knock out mouse lacking “betatrophin” has normal hypertrophy in response to insulin resistance.
Data Reproducibility: Case Study #1 : Betatrophin
Melton et al, Cell 2014 – “Editorial” on betatrophin –states his lab could not confirm the activity they previously reported.
Melton et al PLoS One 2016 – multi lab study further confirming that betatrophic activity in 2013 Cell paper could not be reproduced.
The Bad – Prominent scientist; high profile journal; major claim……….But that major claim is false.
The Good – quick resolution of the error; no disagreement remains; no long term damage was done to the field.
The Questions – What accounts for the mistake having been made? In Melton lab; At the journal that published the paper - Cell.
Data Reproducibility: Case Study #2 : STAP Cells
Stem Cell biology has generated great excitement and therapeutic potential
Approaches to making pluripotent stem cells – Nobel prize to Yamanaka in 2012for technique to make pluripotent stem cells from skin cells using transcription factors.
Data Reproducibility: Case Study #2 : STAP Cells
Stem Cell biology has generated great excitement and therapeutic potential
Approaches to making pluripotent stem cells – Nobel prize to Yamanaka in 2012for technique to make pluripotent stem cells from skin cells using transcription factors.
Jan 2014 – Paper in Nature by Obotaka et al – STAP cells described – “Stimulus-triggered acquisition of pluripotency”
– exposure of skin cells to weak acid makes them pluripotent!(Collaboration between scientists at RIKEN in Japan and Brigham and Womens/HMS)
Data Reproducibility: Case Study #2 : STAP Cells
Stem Cell biology has generated great excitement and therapeutic potential
Approaches to making pluripotent stem cells – Nobel prize to Yamanaka in 2012for technique to make pluripotent stem cells from skin cells using transcription factors.
Jan 2014 – Paper in Nature by Obotaka et al – STAP cells described – “Stimulus-triggered acquisition of pluripotency” – exposure of skin cells to weak acid makes them pluripotent!(Collabortion between RIKEN in Japan and Brigham and Womens/HMS)
Later in 2014 – Many labs unable to reproduce this;- RIKEN conducts investigation, finds Obotaka committed fraud; - Nature retracts paper in July 2014 (6 months after publication)
Data Reproducibility: Case Study #2 : STAP Cells
Stem Cell biology has generated great excitement and therapeutic potential
Approaches to making pluripotent stem cells – Nobel prize to Yamanaka in 2012for technique to make pluripotent stem cells from skin cells using transcription factors.
Jan 2014 – Paper in Nature by Obotaka et al – STAP cells described – “Stimulus-triggered acquisition of pluripotency” – exposure of skin cells to weak acid makes them pluripotent!(Collabortion between RIKEN in Japan and Brigham and Womens/HMS)
Later in 2014 – Many labs unable to reproduce this; - RIKEN conducts investigation, finds Obotaka committed fraud; - Nature retracts paper in July 2014
Sasai, Obotaka advisor and co-author at RIKEN commits suicide in Aug, 2014.
Data Reproducibility: Case Study # 3, 4, n……..
Papers in Cell, Science, Nature etc……. +/- attention by the press
Broad interest in the work, but no clear confirmations or refutationsare published.
Scientists in academia and industry informally discuss their inability to reproducethe work.
Discussion and publication on the topic diminishes, or continues amidst confusion.
How should we view these findings pending further study?: irreproducible, contested, uncertain, or just irrelevant?
Reproducibility of Clinical Research
I have focused on basic science and preclinical research – but the problemof reproducibility is of equal or greater magnitude in clinical research.
Many studies are:
- poorly designed and underpowered - employ faulty statistic approaches and reagents- suffer from investigator biases that are not controlled for- publication bias favors “positive results” – difficulty publishing negative studies
Because these publications involve human subjects, the implications of irreproducibility are even more alarming………..
Research and Publishing Ecosystem
Scientist
Idea
Resources/Institution
Research and Publishing Ecosystem
Scientist
Idea
Resources/Institution
Research Results
Internal discussionsBroader presentationsPossible grant proposal
Research and Publishing Ecosystem
Scientist
Idea
Resources/Institution
Research ResultsSubmission forPublication
Peer Review
Editorial decision
Internal discussionsBroader presentationsPossible grant proposal
Research and Publishing Ecosystem
Scientist
Idea
Resources/Institution
Research ResultsSubmission forPublication
Peer Review
Editorial decision
Internal discussionsBroader presentationsPossible grant proposal
Reject
New submission
Accept!
Reconsider withAdditional work
RejectAccept!
Accept!
It’s Not Finished With Acceptance
Accept!
Part ofScientific Literature
Engagement viaScientific communityOver time
It’s Not Finished With Acceptance
Accept!
Part ofScientific Literature
Engagement viaScientific communityOver time
Impact/meaning ofPublished work confirmed
It’s Not Finished With Acceptance
Accept!
Part ofScientific Literature
Engagement viaScientific communityOver time
Press
Impact/meaning ofPublished work confirmed
It’s Not Finished With Acceptance
Accept!
Part ofScientific Literature
Engagement viaScientific communityOver time
Press
Academic CareersPromotionsGrantsFame and fortune
Impact/meaning ofPublished work confrmed
It’s Not Finished With Acceptance
Accept!
Part ofScientific Literature
Engagement viaScientific communityOver time
Press
Academic CareersPromotionsGrantsFame and fortune
Impact/meaning ofPublished work confirmed
Impact/meaning ofPublished work denied
It’s Not Finished With Acceptance
Accept!
Part ofScientific Literature
Post Publication Review PubPeer, others
Engagement viaScientific communityOver time
Press
Academic CareersPromotionsGrantsFame and fortune
Impact/meaning ofPublished work confirmed
Impact/meaning ofPublished work denied
Irreproducibility assertedAnd/or proven
It’s Not Finished With Acceptance
Accept!
Part ofScientific Literature
Post Publication Review PubPeer, others
Engagement viaScientific communityOver time
Press
Academic CareersPromotionsGrantsFame and fortune
Impact/meaning ofPublished work confirmed
Impact/meaning ofPublished work denied
Irreproducibility assertedAnd/or proven
Investigation
Possible correctionOr retraction
? Sustained Uncertainty
It’s Not Finished With Acceptance
Accept!
Part ofScientific Literature
Post Publication Review PubPeer, others
Engagement viaScientific communityOver time
Press
Academic CareersPromotionsGrantsFame and fortune
Impact/meaning ofPublished work confirmed
Impact/meaning ofPublished work denied
Irreproducibility assertedAnd/or proven
Investigation
Possible correctionOr retraction
? Sustained Uncertainty
It’s Not Finished With Acceptance
Accept!
Part ofScientific Literature
Post Publication Review PubPeer, others
Engagement viaScientific communityOver time
Press
Academic CareersPromotionsGrantsFame and fortune
Impact/meaning ofPublished work confirmed
Impact/meaning ofPublished work denied
Irreproducibility assertedAnd/or proven
Investigation
Possible correctionOr retraction
? Sustained Uncertainty
It’s Not Finished With Acceptance
Not a simple issue – the definition is ambiguous.
Precise replication is the most stringent, but least common approach. This
may be required if claim is important, doubt about finding, high profile,
many labs working in the area (betatrophin, STAP cells).
If finding is robust, it may be seen as “true” even without precise replication;
it is a result that can be productively built on by others.
If finding not robust, precise replication might be required to determine its’
truth, but its’ importance may be seen as diminished.
What does reproducibility of research mean?
- Fully stands the test of time –multiply reproduced and
extended; part of “accepted canon” – (though can always be
reinterpreted or modified by future work)
Irreproducibility of Published Research: The Spectrum.
- Fully stands the test of time –multiply reproduced and
extended; part of “accepted canon” – (though can always be
reinterpreted or modified by future work)
- “Some elements” true, but others, including possibly
important ones, shown to be erroneous based on additional
work. These issues may or may not “change the major
conclusions”, but remain a concern.
Irreproducibility of Published Research: The Spectrum.
- Fully stands the test of time – reproduced and extended; part of
“accepted canon” – (can always be reinterpreted or modified)
- “Some elements” true, but others, including possibly important
ones, seen as erroneous based on additional work. These issues
may or may not “change the major conclusions”, but remain a
concern.
- Viewed as largely or completely untrue – diverse causes:
- honest errors/incompetence
- misconduct, fraud (less common).
Irreproducibility of Published Research: The Spectrum.
- Fully stands the test of time – reproduced and extended; part of
“accepted canon” – (can always be reinterpreted or modified)
- “Some elements” true, but others, including possibly important
ones, seen as erroneous based on additional work. These issues
may or may not “change the major conclusions”, but remain a
concern.
- Viewed as largely or completely untrue – diverse causes:
- honest errors/incompetence
- misconduct, fraud (less common).
Irreproducibility of Published Research: The Spectrum.
Much published work is rarely read, cited, or subject to reproduction, so whether it is ultimately reproducible is unknown.
The Spectrum of Irreproducibility of Published Research
Completelyreproducible
Largelyreproducible
Largelyirreproducible
Completelyirreproducible
How Common?Why?
- We really don’t know – many high estimates - different definitions;
how to assess?
Irreproducibility: How common is it?
- We really don’t know – many high estimates - different definitions;
how to assess?
- Appears to be increasing – as seen by research “watchdogs”, the press,
pharma, funding agencies –
Irreproducibility: How common is it?
Some Data on Reproducibility of Preclinical Research in High Impact Journals
Cost and Categories of Irreproducible Preclinical Research
New “Science of Reproducibility”, and New Watchdogs
Ivan Oransky
- more common in high profile journals
( ? more attention to those papers)
Are Some Journals More Irreproducible?
- number of retractions is clearly increasing over past 15 years –
does this represent more scrutiny, better tools to uncover
manipulation vs change in underlying rate?
Irreproducibility: Retractions Rising
? Tip of the iceberg
- Poor training/ experimental methodology – design, statistics, inference;
Observer bias, lack of double blind or internal replication
- Poorly characterized reagents – antibodies (? Half of analytic antibodies
give false/non specific results), inhibitors, cell lines
- Complex, interdisciplinary collaborations diminish understanding of
project elements by principal investigator (“I didn’t understand that
part”)
- Poor oversight/mentorship re: integrity of data and its presentation
(selective reporting)
- Deficiencies within the peer review/publishing ecosystem
Irreproducibility: Causes (1)
- Inappropriate response to external incentives –
grants, promotions, fame, financial gain
“cutting corners”, selective data reporting, etc
blend into or lead to more explicit manipulation and misconduct.
Irreproducibility: Causes (2)
- Ethical lapses sociopathy -
The most extreme cases can be hard to identify --- results “too good to be true”Be skeptical, don’t hesitate to have others confirm -
How Do Journals Contribute to the Problem?
Criteria for publication – impact more than the truth.
Excessive interest in Novelty/impact/Buzz
Lack of interest by publisher in confirmation, refutation, or important extensions of published work -- “publication bias”
Editorial requests for more and specific data – dangerous incentives to provide “required experimental results”, tempting susceptible scientists.
Changes in Scientific Publishing That Might Enhance Reproducibility of Published Work
(and improve the quality of peer review/editorial decision-making)
Changes to Peer Review –
- Make reviews of papers publically available and citable
- Identify reviewers to authors and community
- All versions of accepted papers kept online
- Availability of raw data for independent analysis
Goal : Accountability; higher quality; less bias
What can academia do about the problem?
- Accept that it is a problem, and get serious about addressing it.
- Some successful scientists and institutions have too little interest
in this issue – they think the system serves them well.
- Enhance access to and quality of training in experimental
design/stats/research and publication ethics.
What can academia do about the problem?
- Attempt to modify the culture of research. (This is hard to do).
More rewards/respect for those who produce reproducible
results, and those who take responsibility when their work does not
meet these standards; fewer rewards/respect for those who do not.
- culture responds more to actions, than to words.
- Reflect this in approach to academic appointments, promotions,
awards, etc -
( develop “Reproducibility index” to complement the H index……… )
Rx for Irreproducibility: The Ecosystem
Scientist andResearch culture
Institutional Culture andExpectations
Funders of Research
Biotech andPharma
PublishingEcosystem
The Pressand
Social Media
Two main goals: 1) Aligning incentives 2) Accountability
May Your Research be Both Important and Reproducible!
Thank You for inviting me to the lovely city of Thessaloniki!!
Research and Publishing EcosystemScientist
Idea
Resources/Institution
Research Results Submission forPublication
? Archive
Peer Review
Editorial decision
Internal discussionsBroader presentationsPossible grant proposal
Reject
New submissionAccept!
Reconsider withAdditional work
RejectAccept!
Part ofScientific Literature Post Publication Review
PubPeer, others
Engagement viaScientific communityOver time
Press
Academic CareersPromotionsGrantsFame and fortune
Impact/meaning ofPublished work secured
Impact/meaning of
Published work deniedIrreproducibility assertedAnd/or proven
Investigation
Possible correctionOr retraction
Sustained Uncertainty
+2000 journals400,000 pub articles1.3 M submitted
Accept!
Part ofScientific Literature
Post Publication Review PubPeer, others
Engagement viaScientific communityOver time
Press
Academic CareersPromotionsGrantsFame and fortune
Impact/meaning ofPublished work secured
Impact/meaning ofPublished work denied
Irreproducibility assertedAnd/or proven
Investigation
Possible correctionOr retraction
? Sustained Uncertainty
It’s Not Over With Acceptance
- We really don’t know – many estimates - different definitions;
how to assess?
- Perception that it is increasing – expressed by watchdogs, press,
pharma, funders – Amgen paper on preclinical cancer research
as example
- ? more common in high profile journals vs more attention to
those papers (many papers rarely read, referenced).
- number of retractions is clearly increasing over past 15 years –
does this represent more scrutiny, better tools to uncover
manipulation vs change in underlying rate?
Irreproducibility: How common is it?
Research and Publishing Ecosystem
The Publishing Ecosystem: The Players
Publishers : Scientific/Professional Societies; For profit publishers; many new models (e-life)
Editors – scientific leaders vs professional editors
Reviewers – domain experts; unpaid
Peer review models –Closed (anon reviewers, hidden reviews, final version published) vs Open ( reviewers identified, reviews published, multiple versions)
Financial models – subscription vs open access with fees to publish
Print vs digital
- Can be slower than ideal to publish results for scientific community to
evaluate.
- Publication bias – disinterest in negative data, confirmation, “incomplete
stories” ; exaggerated interest in claims of novelty
- May favor the work of some scientists over others independent of the
quality/importance of the work – “old boys network”
- Expensive; inadequate public access
What are some flaws of the current publishing ecosystem?
- May favor the work of some scientists over others independent of the
quality/importance of the work – “old boys network”
- Does not optimize quality and value of the review process to the authors or
the reviewers.
- At a time of digital access and interconnectedness, it should be possible to
disrupt the current system to produce outcomes more valuable to the
scientists, their institutions, and funders – and eventually the public at large.
What are some flaws of the current publishing ecosystem?