january 13, 2016 niso webinar: ensuring the scholarly record: scholarly retractions, scientific...
TRANSCRIPT
Life after publication:
Reproducibility challenges
and the role of publishersVéronique Kiermer, Executive Editor
PLOS
January 2016
Retractions, reproducibility and the
integrity of the scientific record
Overview
• Retractions:
• Challenges and limitations from an editorial perspective
• Irreproducibility:
• Underlying causes
• What can editors and publishers do?
Retractions
• Retraction of a research article is a complete and permanent removal from the scientific record
• Although an article remains accessible to readers, it should no longer be cited.
• Reasons for retraction:
• Conclusions cannot be relied upon and are no longer supported (invalid results)
• Serious breach of research or publication ethics
Retraction trends
Van Noorden, Nature 478, 26-28 (2011)
In same period, volume of papers increased by 44%
Why are papers retracted?
Van Noorden, Nature 478, 26-28 (2011)
Why are retraction notices opaque?
Ideal retraction (and correction) notices:• Transparent about scientific facts and responsibility• Distinguish misconduct from honest errors• Result from careful and unbiased investigation• Are published without delay
In reality: • Cases are incredibly complex• It takes time and resources to investigate• Misconduct finding requires the determination of intent.• Journals do not have the means and the authority to
investigate misconduct cases• Journals rely on institutional investigations• Occasional legal issues
Journals and Universities: two perspectives on the same problem
Journals
• Focus on the integrity of the
scientific record
• Seek to rapidly correct the
scientific facts
• Can request original data,
explanations, experiments
Universities
• Focus on determining
responsibility and guilt
• Seek to take corrective
actions
• Can gain access to
records, lab notebooks,
sequester evidence
September 2009Elizabeth Wager, Virginia Barbour, Steven Yentis, Sabine Kleinert on behalf of COPE
Council
“The main purpose of retractions is to correct the literature and ensure its integrity rather than
to punish authors who misbehave.”
RETRACTION GUIDELINES
(For authorship disputes occurring before publication, see the relevant COPE flowcharts. http://publicationethics.org/files/u2/04A_Author_Add_Submitted.pdf and http://publicationethics.org/files/u2/04B_Author_Remove_Submitted.pdf)
Can authors dissociate themselves from a retracted publication?
If retraction is due to the actions of some, but not all, authors of a publication, the notice of retraction should mention this. However, most editors consider that authorship entails some degree of joint responsibility for the integrity of the reported research so it is not appropriate for authors to dissociate themselves from a retracted publication even if they were not directly culpable of any misconduct.
Are there grounds for legal proceedings if an author sues a journal for retracting, or refusing to retract, a publication?
Authors who disagree with a retraction (or whose request to retract a publication is refused) sometimes threaten journal editors with legal action. Concern over litigation can make editors reluctant to retract articles, especially in the face of opposition from authors.
Journals’ instructions for authors should explain the retraction procedure and describe the circumstances under which articles might be retracted. This information should be incorporated (e.g. by references) into any publishing agreements and brought to the authors’ attention. However, even if the publishing agreement or journal instructions do not set out specific conditions for retraction, authors usually would not have grounds for taking legal action against a journal over the act of retraction if it follows a suitable investigation and proper procedures.
However, legal advice may be helpful to determine appropriate wording for a notice of retraction or expression of concern to ensure that these are not defamatory or libellous. Nevertheless, retraction notices should always mention the reason(s) for retraction to distinguish honest error from misconduct.
Whenever possible, editors should negotiate with authors and attempt to agree a form of wording that is clear and informative to readers and acceptable to all parties. If authors consent to the wording of a retraction statement, this provides defence against a libel claim. However, prolonged negotiations about wording should not be allowed to delay the publication of a retraction unreasonably and editors should publish retractions even if consensus cannot be reached.
WWW.PUBLICATIONETHICS.ORG
Process and best practices
• Retractions have their own DOI.
• Permanent bi-directional linking & clear marking of the paper – syndicate to indexers.
• For corrections, clear indication if the paper was republished
CrossRef industry-wide initiative to provide a standard way for readers to locate the most up-to-date version of an article.
PLOS has adopted
Irreproducibility
PloS Medicine 2005doi: 10.1371/journal.pmed.0020124
Nature 2012doi:10.1038/483531a
Discussing irreproducibility
ü We are not talking about fraud.
ü We acknowledge that reasonable conclusions derived from legitimate observations can be disproved by subsequent knowledge and technology advancements.
ü We distinguish: replication ≠ generalization …and we draw conclusions accordingly.
ü We must talk about and reduce irreproducibility due poor reporting, experimental and analytical design, …
META-RESEARCH ARTICLE
Reproducible Research Practices andTransparency across the BiomedicalLiteratureShareen A. Iqbal1☯, Joshua D. Wallach2,3☯, Muin J. Khoury4,5, Sheri D. Schully4, John P.A. Ioannidis2,3,6,7*
1 Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, Georgia, UnitedStates of America, 2 Department of Health Research and Policy, Stanford School of Medicine, Palo Alto,California, United States of America, 3 Meta-Research Innovation Center at Stanford, Stanford University,Stanford, California, United States of America, 4 Division of Cancer Control and Population Sciences,National Cancer Institute, National Institutes of Health, Bethesda, Maryland, United States of America,5 Office of Public Health Genomics, Centers for Disease Control and Prevention, Atlanta, Georgia, UnitedStates of America, 6 Stanford Prevention Research Center, Department of Medicine, Stanford University,Stanford, California, United States of America, 7 Department of Statistics, Stanford University School ofHumanities and Sciences, Stanford, California, United States of America
☯ These authors contributed equally to this work.* [email protected]
AbstractThere is a growing movement to encourage reproducibility and transparency practices inthe scientific community, including public access to raw data and protocols, the conduct ofreplication studies, systematic integration of evidence in systematic reviews, and the docu-mentation of funding and potential conflicts of interest. In this survey, we assessed the cur-rent status of reproducibility and transparency addressing these indicators in a randomsample of 441 biomedical journal articles published in 2000–2014. Only one study provideda full protocol and none made all raw data directly available. Replication studies were rare(n = 4), and only 16 studies had their data included in a subsequent systematic review ormeta-analysis. The majority of studies did not mention anything about funding or conflicts ofinterest. The percentage of articles with no statement of conflict decreased substantiallybetween 2000 and 2014 (94.4% in 2000 to 34.6% in 2014); the percentage of articles report-ing statements of conflicts (0% in 2000, 15.4% in 2014) or no conflicts (5.6% in 2000, 50.0%in 2014) increased. Articles published in journals in the clinical medicine category versusother fields were almost twice as likely to not include any information on funding and to haveprivate funding. This study provides baseline data to compare future progress in improvingthese indicators in the scientific literature.
Author SummaryThere is increasing interest in the scientific community about whether published researchis transparent and reproducible. Lack of replication and non-transparency decreases the
PLOS Biology | DOI:10.1371/journal.pbio.1002333 January 4, 2016 1 / 13
OPEN ACCESS
Citation: Iqbal SA, Wallach JD, Khoury MJ, SchullySD, Ioannidis JPA (2016) Reproducible ResearchPractices and Transparency across the BiomedicalLiterature. PLoS Biol 14(1): e1002333. doi:10.1371/journal.pbio.1002333
Academic Editor: David L Vaux, Walter and ElizaHall Institute of Medical Research, AUSTRALIA
Received: October 13, 2015
Accepted: November 19, 2015
Published: January 4, 2016
Copyright: This is an open access article, free of allcopyright, and may be freely reproduced, distributed,transmitted, modified, built upon, or otherwise usedby anyone for any lawful purpose. The work is madeavailable under the Creative Commons CC0 publicdomain dedication.
Data Availability Statement: All relevant data arewithin the paper and its Supporting Information files.All authors had full access to all of the data (includingstatistical reports and tables) in the study and cantake responsibility for the integrity of the data andaccuracy of the data analysis.
Funding: The authors received no specific fundingfor this work. The Meta-Research Innovation Centerat Stanford (METRICS) is supported by a grant fromthe Laura and John Arnold Foundation.
Competing Interests: The authors have declaredthat no competing interests exist.
PLOS Biology | DOI: 10.137/journal.pbio.1002333 | Jan 4, 2016
Random sample: 441 biomedical journal articles 2000-20141 study provided a full protocolNone made all raw data directly availableReplication studies were rare (n=4)The majority of studies did not mention anything about funding or conflict of interest
Reporting
Experimental bias
Fig 1. (A) Prevalence of reporting of randomisation, blinded assessment of outcome, sample sizecalculation, and conflict of interest in 146 publications describing in vivo research identified throughrandom sampling from PubMed; change in prevalence of (B) randomisation, (C) blinded assessment
PLOS Biology | DOI:10.1371/journal.pbio.1002273 October 13, 2015 4 / 12
Fig 1. (A) Prevalence of reporting of randomisation, blinded assessment of outcome, sample sizecalculation, and conflict of interest in 146 publications describing in vivo research identified throughrandom sampling from PubMed; change in prevalence of (B) randomisation, (C) blinded assessment
PLOS Biology | DOI:10.1371/journal.pbio.1002273 October 13, 2015 4 / 12
Fig 1. (A) Prevalence of reporting of randomisation, blinded assessment of outcome, sample sizecalculation, and conflict of interest in 146 publications describing in vivo research identified throughrandom sampling from PubMed; change in prevalence of (B) randomisation, (C) blinded assessment
PLOS Biology | DOI:10.1371/journal.pbio.1002273 October 13, 2015 4 / 12
Fig 1. (A) Prevalence of reporting of randomisation, blinded assessment of outcome, sample sizecalculation, and conflict of interest in 146 publications describing in vivo research identified throughrandom sampling from PubMed; change in prevalence of (B) randomisation, (C) blinded assessment
PLOS Biology | DOI:10.1371/journal.pbio.1002273 October 13, 2015 4 / 12
Macleod et al., PLOS Biology | DOI: 10.1371/journal.pbio.1002273 | Oct 12, 2015
Fig 1. (A) Prevalence of reporting of randomisation, blinded assessment of outcome, sample sizecalculation, and conflict of interest in 146 publications describing in vivo research identified throughrandom sampling from PubMed; change in prevalence of (B) randomisation, (C) blinded assessment
PLOS Biology | DOI:10.1371/journal.pbio.1002273 October 13, 2015 4 / 12
Randomisation Blinded assessment Competing interest
Random sample (n=146) describing in vivo research:Limited reporting of measures to reduce the risk of bias
Analytical bias
PERSPECTIVE
The Extent and Consequences of P-Hacking inScienceMegan L. Head1*, Luke Holman1, Rob Lanfear1,2, Andrew T. Kahn1, Michael D. Jennions1
1 Division of Evolution, Ecology and Genetics, Research School of Biology, Australian National University,Acton, Canberra, Australia, 2 Department of Biological Sciences, Faculty of Science, Macquarie University,North Ryde, New SouthWales, Australia
AbstractA focus on novel, confirmatory, and statistically significant results leads to substantial biasin the scientific literature. One type of bias, known as “p-hacking,” occurs when researcherscollect or select data or statistical analyses until nonsignificant results become significant.Here, we use text-mining to demonstrate that p-hacking is widespread throughout science.We then illustrate how one can test for p-hacking when performing a meta-analysis andshow that, while p-hacking is probably common, its effect seems to be weak relative to thereal effect sizes being measured. This result suggests that p-hacking probably does notdrastically alter scientific consensuses drawn from meta-analyses.
IntroductionThere is increasing concern that many published results are false positives [1,2] (but see [3]).Many argue that current scientific practices create strong incentives to publish statisticallysignificant (i.e., “positive”) results, and there is good evidence that journals, especially presti-gious ones with higher impact factors, disproportionately publish statistically significant results[4–10]. Employers and funders often count papers and weigh them by the journal’s impactfactor to assess a researcher’s performance [11]. In combination, these factors create incentivesfor researchers to selectively pursue and selectively attempt to publish statistically significantresearch findings.
There are two widely recognized types of researcher-driven publication bias: selection(also known as the “file drawer effect”, where studies with nonsignificant results have lowerpublication rates [7]) and inflation [12]. Inflation bias, also known as “p-hacking” or “selectivereporting,” is the misreporting of true effect sizes in published studies (Box 1). It occurs whenresearchers try out several statistical analyses and/or data eligibility specifications and thenselectively report those that produce significant results [12–15]. Common practices that lead top-hacking include: conducting analyses midway through experiments to decide whether tocontinue collecting data [15,16]; recording many response variables and deciding which to re-port postanalysis [16,17], deciding whether to include or drop outliers postanalyses [16], ex-cluding, combining, or splitting treatment groups postanalysis [2], including or excludingcovariates postanalysis [14], and stopping data exploration if an analysis yields a significantp-value [18,19].
PLOS Biology | DOI:10.1371/journal.pbio.1002106 March 13, 2015 1 / 15
OPEN ACCESS
Citation: Head ML, Holman L, Lanfear R, Kahn AT,Jennions MD (2015) The Extent and Consequencesof P-Hacking in Science. PLoS Biol 13(3): e1002106.doi:10.1371/journal.pbio.1002106
Published: March 13, 2015
Copyright: © 2015 Head et al. This is an openaccess article distributed under the terms of theCreative Commons Attribution License, which permitsunrestricted use, distribution, and reproduction in anymedium, provided the original author and source arecredited.
Funding: Funding for this research was provided byAustralian Research Council Grants awarded to MDJ,RL and LH. The funders had no role in study design,data collection and analysis, decision to publish, orpreparation of the manuscript.
Competing Interests: The authors have declaredthat no competing interests exist.
Abbreviations: NHST, Null hypothesissignificance testing.
PLOS Biology | DOI: 10.1371/journal.pbio.1002106 | Mar 13, 2015
Underlying issues
• experimental design• statistics literacy• data presentation
• data management• reagents validity
• pressure to publish• publication bias • replications and
refutations not pursued
training laboratory managementleadership & mentoringsize of laboratories
infrastructureoversight and compliancequality assurance
incentives for rigor, professionalism and good laboratory leadership
REMEDIES
researcherlab PI
university &research institution
university &research institution
scientific community
ISSUES
What can journals do?
Raiseawareness
& promoteresearch
Joint initiatives
Joint initiatives
Promote reporting standards
PLOS requires adherence to multiple reporting standards:- CONSORT, TREND, STARD, STROBE- PRISMA- MIAMESupports specialist standards
PLOS data policy
PLOS journals require authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception.
When submitting a manuscript online, authors must provide a Data Availability Statement describing compliance with PLOS's policy. If the article is accepted for publication, the data availability statement will be published as part of the final article.
Since March 3, 2014
GuidanceGeneral guidelines: http://journals.plos.org/plosone/s/data-availabilityplos
EDITORIAL
PLOS Genetics Data Sharing Policy: In Pursuitof Functional UtilityGregory S. Barsh1,2*, Gregory M. Cooper2, Gregory P. Copenhaver3, Greg Gibson4, MarkI. McCarthy5,6,7, Hua Tang1, Scott M. Williams8¤
1 Department of Genetics, Stanford University School of Medicine, Stanford, California, United States ofAmerica, 2 HudsonAlpha Institute for Biotechnology, Huntsville, Alabama, United States of America,3 Department of Biology, The University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, UnitedStates of America, 4 Center for Integrative Genomics, School of Biology, Georgia Institute of Technology,Atlanta, Georgia, United States of America, 5 Wellcome Trust Centre for Human Genetics, University ofOxford, Oxford, United Kingdom, 6 Oxford Centre for Diabetes, Endocrinology & Metabolism, University ofOxford, Oxford, United Kingdom, 7 Oxford NIHR Biomedical Research Centre, Churchill Hospital, Oxford,United Kingdom, 8 Department of Genetics, Geisel School of Medicine, Dartmouth College, Hanover, NewHampshire, United States of America
¤ Current address: CaseWestern Reserve University School of Medicine, Cleveland, Ohio, United States ofAmerica* [email protected]
About a year ago, PLOS implemented a new process intended to further the overarching prin-ciple that data used in the work we publish should be accessible and reusable. The motivationgoes hand-in-hand with both our open access ethos and the scientific method itself: the validityof a conclusion depends on the ability to reproduce the underlying results.
In theory, PLOS’ new data policy, in which “all data underlying the findings described[must be] fully available,” is not so new; in practice, though, it may be perceived as burden-some, complicated, and/or inefficient. The policy and its purpose has been discussed exten-sively among PLOS Genetics Editorial Board members with regard to its potential impact oneditors, authors, the community, and research subjects.
The purpose of this editorial is 2-fold: to acknowledge and discuss aspects of the “data-shar-ing” process that are especially challenging, and to provide additional clarification and guid-ance in the context of a few scenarios that are especially relevant, all from the perspective ofworking scientists who read, evaluate, and contribute to research based on genetics and geno-mics. We also suggest a way forward that builds on what already works at PLOS: consultingwith multiple stakeholders whose interests intersect to develop consensus.
An issue that is often problematic for PLOS Genetics authors is the sheer volume of datagenerated by large-scale phenotyping and/or genotyping studies. Whether from a confocalmicroscope, a radiofrequency detector in an MRI, or a CCD in a DNA sequencing instrument,processing, filtering, and compression of digital data is inherent across many areas of modernbiology. It is often neither practical nor wise to archive and distribute the primary output ofdigital detectors; indeed, the question of what constitutes “raw data” is a moving target.
A second problematic issue arises from ethical concerns associated with human researchsubjects. The potential to identify research subjects based on genomic information has receivedconsiderable attention and has fostered the development of controlled access mechanisms, inwhich researchers must seek approval from data access committees. As with any new set of reg-ulations, there is a risk of creating more problems than are solved; from an editorial perspec-tive, we recognize that authors must commit significant resources to deposit data into
PLOSGenetics | DOI:10.1371/journal.pgen.1005716 December 14, 2015 1 / 5
OPEN ACCESS
Citation: Barsh GS, Cooper GM, Copenhaver GP,Gibson G, McCarthy MI, Tang H, et al. (2015) PLOSGenetics Data Sharing Policy: In Pursuit of FunctionalUtility. PLoS Genet 11(12): e1005716. doi:10.1371/journal.pgen.1005716
Published: December 14, 2015
Copyright: © 2015 Barsh et al. This is an openaccess article distributed under the terms of theCreative Commons Attribution License, which permitsunrestricted use, distribution, and reproduction in anymedium, provided the original author and source arecredited.
Funding: No specific funding was received for thisarticle.
Competing Interests: GSB and GPC are Editors-in-Chief of PLOS Genetics. GG, HT, and SMW areSection Editors for the Natural Variation Section forPLOS Genetics. GMC is an Associate Editor andMIM is a former Associate Editor for PLOS Genetics.
PLOS External Data Advisory GroupPLOS Data Lead: Emma GanleyAcademic chair: Phil Bourne
Discipline specific guidelines:
On January 7, 2016, a coalition of publishers signed an Open Letter committing to start requiring ORCID IDs in 2016.
1. Implementing best practices for ORCID collection and auto-update of ORCID records upon publication
2. Require ORCID IDs for corresponding authors and encourage for co-authors
Credit and accountability for authors
A simple taxonomy of research contributions developed under the leadership of CASRAI and NISO.
- Includes but not limited to traditional authorship roles
- Makes contributions machine-readable and portable
- Meant to inspire development: Mozilla badges, VIVO-ISF ontology, JATS integration, ORCID integration
Editorial and peer review evaluation
Editorial office:• Trial registration• Data deposition• Reporting guidelines• Ethical approval• Competing interests• Financial disclosures• Permissions• Plagiarism• Image integrity
Peer reviewers:• Methodology and
experimental design• Analysis• Statistics• Conclusions• Ethics
Limitations: - Science has become more cross-disciplinary- Confidential peer review can show biases- Paper should live on after publication- Still not enough incentives to publish all results
Peer review innovations
Missing pieces:
• Publication of all valid,
technically sound
research
• Transparent reporting
• Data, code and reagent
sharing
• Robust post-publication
discussions among
community
• Credit for all contributions
Thank you for listening!