irreproducible results - portal · results of reproducibility study for 508 papers 402 9...
TRANSCRIPT
Coffee & Bit(e)sTHE COFFEE LECTURES FOR SCIENTISTS
UNIVERSITY LIBRARY BERNAline Frank, Nuria Plattner, Michael Horn, and Silvan Christenwww.unibe.ch/ub/sciencelibrary
Dr. Nuria Plattner
Irreproducible results in
scientific articlesCan we trust published literature?
2
Scientific Method based on reproducibilityImportance of reproducibility
• A scientific fact can only be established if a reproducibleprocedure to verify it can be defined
• Research topic: challenges for defining reproduciblemethods and documentation differ for different areas
• The reproducibility problem exists in all areas
• Established vs. new topics: the more established aresearch area, the better defined and standardized itsprocedures
https://3c1703fe8d.site.internapcdn.net/newman/gfx/news/hires/2016/howscientist.png
3
Recent concerns about reproducibilityIs there a reproducibility crisis?
M. Baker, Nature 533, 452, (2017)
• Nature survey, 1576researchers
• 52% agree that thereis a significant crisis inreproducibility
• <31% think that non-reproducible resultsare wrong
• Most scientist still trustpublished literature
4
Trends in paper retractionCrisis Indicators
A) Retracted articles for specific causes by retraction year
B) Percentage of articles retracted for fraud or suspected fraud by publication year
• Only papers with significantflaws are retracted
• Minor corrections usually addedto paper as erratum (if anything)
• Main causes for retraction: fraudor scientific mistake (conceptual)
F. C. Fang, R. G. Steen, and A. Casadevall, Proc. Natl. Accad. Sci. 109, 17028-17033, (2012)
5
Scientific result
Publication process and lifecycleHow are irreproducible results detected?
ManuscriptPeer review
Experimental results/observations: only consistency check
Mathematical proofs: full check
Reproductionattempt of result
Nobody interested
Check forown mistakescontact authorCorrection/
Retraction
Prepare publicationabout new finding
?
Scientific publication
6
Statistical facts about scientific publicationsReasons for reproducibility problems
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3999612/
• Scientists spend time publishing as much aspossible, rather than spending time developingresearch
• No time for confirmation of experiments
• Limited time for peer-review
• Only 45% of articles published in top 4500scientific journals are cited within first 5 years
• Only 42% of the papers receive more than onecitation, 5-25% of these are self-citation
7
Selective reporting of resultsExamples for lack of reproducibility
• Selective reporting can take place in various ways
• There can be a fine line between selective reporting and scientific fraud
Different forms of selective reporting[1]
[1] Timothy H. Parker et al., Trends Ecol. Evol., 31, 711-719, (2016)
Examples of selective reporting[1]
8
Missing documentation/codeExamples for lack of reproducibility
• Sharing for repeatability is essential toensure that other researchers canevaluate results based on accurate andcomplete evidence
• Repeatability study 2015 examinsextent to which computer systemsresearchers share their source code andcode builds
• This is referred to as weak repeatability
http://reproducibility.cs.arizona.edu/
Results of reproducibility study for 508 papers
402
9
Experiments with limited reproducibilityExamples for lack of reproducibility
• Success rates for new development projects in Phase II trials have fallen from 28% to 18%in recent years
• Limitations ofpredictivity
• Validity of targets
F. Prinz, T. Schlange and K. Asadullah, Nat. Rev. Drug Discovery 10, 712, (2011)
Bayer study: reproducibility of preclinical trials
c) relationship of publisheddata to in-house data
a) characterization oftargets
b) approaches used toreprocuded data
10
Scientific misconductExamples for lack of reproducibility
• Nr. 1) Yoshitaka Fujii, anesthesiology researcher, fabricated clinical study data in at least 169 papers
• 1993: falsified data publishing started• 2000: first publication suspecting fraud• 2012: full investigation
• Nr. 10) Jan Hendrik Schön, physicist at Bell labs; high-impact papers retracted after failure toreproduce superconductivity in organic materials
• 2000: first fraudulent result published• 2001: first suspicion of fraud• 2002: report of scientific misconduct released
https://www.sciencemag.org/news/2018/10/what-massive-database-retracted-papers-reveals-about-science-publishing-s-death-penalty
Medicine
Psychology
Computer Science
Medicine
Medicine
Chemistry
Physics
Biology
Accounting (Economy)
Medicinal Chemistry
11
Retraction watch databaseHow to keep track of paper retractions?
https://retractionwatch.com
12
Retraction watch databaseHow to keep track of paper retractions?
https://retractionwatch.com
13
Possible solutionsHow to improve reproducibility?
• Important: detailed description ofresults and methods
• sharing of data and code, openscience
• More fundamental changes: structuralincentives need to be changed
BORIS Repository
https://boris.unibe.ch/
Coffee & Bit(e)sTHE COFFEE LECTURES FOR SCIENTISTS
UNIVERSITY LIBRARY BERNAline Frank, Nuria Plattner, Michael Horn, and Silvan Christenwww.unibe.ch/ub/sciencelibrary
Questions?Thanks for your attention