how do we know what we know?
TRANSCRIPT
How Do We Know What We Know?
Ivan Oransky, MDExecutive Editor, Reuters HealthCo-Founder, Retraction Watch
@ivanoransky
UTPABioethics: Creating and Challenging Knowledge in Health
Edinburg, Texas, April 2013
Retractions on the Rise
-The Wall Street Journal
How Often Are Studies Wrong?
“Winner Takes All” Incentives
Scientific American, August 2012
“Winner Takes All” Incentives
“The winner-take-all aspect of the priority rule has its drawbacks, however. It can encourage secrecy, sloppy practices, dishonesty and an excessive emphasis on surrogate measures of scientific quality, such as publication in high-impact journals.”
-- Fang and Casadevall, Scientific American
Anonymous Whistleblowers Step Up
http://www.labtimes.org
Blogs Get Aggressive
http://abnormalscienceblog.wordpress.com/
Blogs Get Aggressive
Blogs Get Aggressive
http://md-anderson-cc.blogspot.com
Blogs Get Aggressive
http://www.science-fraud.org/
Journals are Listening
Retraction Watch
Post-Publication Peer Review
Nature (22 Dec 2011) doi:10.1038/480449a
Post-Publication Peer Review
http://crossref.org/
Post-Publication Peer Review
Post-Publication Peer Review
http://precedings.nature.com/
Post-Publication Peer Review
Alt Metrics
http://total-impact.org/
Alt Metrics
http://sciencecard.org/
How Often Are Medical Studies Wrong?
Ioannidis JPA. PLoS Med 2005; 2(8): e124
How Often Are Medical Studies Wrong?
Does The Literature Reflect Reality?
Does The Literature Reflect Reality?
Publish a trial that will bring US$100,000 of profit or meet the end-of-year budget by firing an editor.
-- Former BMJ editor Richard Smith
Positive Publication Bias
Positive Publication Bias
“The overall frequency of positive supports has grown by over 22% between 1990 and 2007, with significant differences between disciplines and countries.”
“…the strongest increase in positive results was observed in disciplines—like Clinical Medicine, Pharmacology & Toxicology, Molecular Biology”
Fanelli, Scientometrics 2012.
Publish All Data?
Publish All Data?
www.AllTrials.net
FDA: “Black-or-White Approval”
FDA: “Black-or-White Approval”
“…abandon the current black-or-white approval process in favor of an incremental, conditional one. In such a process, drugs could be provisionally approved after promising early-stage data, with the FDA retaining the option to revoke that approval later on, should unexpected data come to light.”
“A ‘conditional approval’ approach would grant limited marketing authorization to new drugs after successful Phase II trials.”
http://www.manhattan-institute.org/
Confirmation Biases
“Facts do not accumulate on the blank slates of researchers' minds and data simply do not speak for themselves. Good science inevitably embodies a tension between the empiricism of concrete data and the rationalism of deeply held convictions.”
“…awareness of the systematic errors that can occur in evaluative processes may facilitate the self regulating forces of science and help produce reliable knowledge sooner rather than later.”
-- Kaptchuk, BMJ, 2003;326:1453–5
Believers vs. Snails
-- Kaptchuk, BMJ, 2003;326:1453–5
Contact/Acknowledgements
http://retractionwatch.com
@ivanoransky
Thanks to Nancy Lapid, Reuters Health