awhiles study day 13 may 2011 julie browne external relations manager, wales deanery, pgmde...
TRANSCRIPT
AWHILES Study Day13 May 2011
Julie BrowneExternal Relations Manager,
Wales Deanery, PGMDEUniversity of Cardiff
“Published research: how do you know if it’s any good?”
…or
“The role of the editorial office in safeguarding the integrity and
academic rigour of the scholarly archive, with particular reference to
the medical education literature”
What I’d like to cover today
• The role of journals in safeguarding quality
• Medical education journals
• Peer review – what is it?
• Peer review – how is it done?
• Peer review – why do it?
• Publication ethics
Meet Jamie …
“How do you know if a published paper is likely to be
reliable?”
Jamie again …
“What are the characteristics of a high-quality journal?”
Quality in journals
• Metrics
• People
• Content
• Systems
Leg 1: Metrics• Citations (including self-citation)• Indexes• Rejection rates• Speed of peer review/acceptance to
publication• Impact factor/immediacy
ISI Web of Knowledge: Journal Citation Reports
0
0.5
1
1.5
2
2.5
3
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Impact Factors in Medical Education Journals by Year
Impa
ct F
acto
r
Acad Med
AHSE* from 2003
Med Educ
Med Teacher
Teach Learn Med* from 1998
J Cont Educ Health* from 2007
Trend spotting: Impact factor
Leg 2: People
– Editorial board, editor, editorial staff– Publisher– Learned society– Peer reviewers– Authors
Leg 3: Content
• High quality papers (goes without saying – we’ll discuss this further in a minute!)
• Letters and commentaries sections (post-hoc peer review)
• Editorials and guidance on scope and mission
• Retractions, corrections, errata
• Participation in journalology research
Leg 4: Editorial Systems• Transparent processes, clearly announced – how we
do things, what you can expect from us• Published policies: e.g. ethics, peer review, handling
of complaints, handling of editors’ papers• Membership of COPE or other editors’ group• An ethics committee• A statement regarding editorial independence
(e.g. “All content in Academic Medicine reflects the views of the authors and does not reflect the official policy of the AAMC unless clearly specified.” “The BMA grants editorial freedom to the editor of the BMJ. The views expressed in the journal are those of the authors and may not necessarily comply with BMA policy.”)
But, above all…
A rigorous, effective and fair peer review system
“The purpose of peer review is to keep egg off authors’ faces”
Drummond Rennie (JAMA)
Completion of research
Writing and preparation
Submission
Internal review
Decision Revision
Resubmission
Re-reviewAcceptance
Publication
Rejection
Rejection
The review process
External review
Common reasons for rejection• Did not address an important scientific issue• Research question not clearly stated• Has been done before• Results are unsurprising/does not add enough new information to
be interesting• Inappropriate study design/methodological weaknesses/methods
not described clearly enough• Sample size too small/response rate too low• Incorrect or unjustified conclusions• Statistical analysis incorrect or inappropriate• Wrong journal/readership; would not interest our readers• Too badly written to understand
But sometimes something murky is lurking under the
surface….
• Fabrication: invention of data or cases• Falsification: wilful distortion of data including
failure to report missing or negative data• Plagiarism: copying of ideas, data or words
without attribution• Failing to get consent from an ethics committee
for research and/or conducting research in humans without informed consent
• Misreporting authorship contributions• Not revealing conflicts of interest
Murky things: misconduct
Peer review to the rescue?!
More from Jamie …
“How widespread is research and publication misconduct?”
“How much does the peer review system catch?”
Things inevitably slip through …
Some big cases (1)
Jon Sudbø
In January 2006 it was revealed that a paper by Norwegian researcher Jon Sudbø published in The Lancet in October 2005 was based upon fraudulent patient data. The article had suggested that some types of NSAIDs diminish the risk of oral cancer in smokers. However, it turned out that he had made up all the patient data; a quarter of all the subjects in The Lancet study had the same birthday!
Some big cases (2)Andrew Wakefield
1996 A paper by Wakefield and colleagues published in The Lancet suggested that there was a connection between the MMR vaccine and autism and colitis. 1998 No subsequent research was able to reproduce his findings. The paper was retracted. 2001 Immunisation rates dropped from 92% to 73%, almost certainly leading to a rise in mortality.2010 Wakefield was struck off by GMC for over 30 charges of misconduct including four counts of dishonesty. Wakefield continues to defend his reputation.
Malcolm Pearce
1994: Obstetrician Malcolm Pearce claimed in BJOG to have successfully relocated an ectopic pregnancy, with a healthy subsequent birth. A junior doctor at St George’s blew the whistle. The patient had never existed.
Some big cases (3)
“OK, so these are big cases.But does the little stuff really matter
that much …?”
The journalology literature
• 56% of newly appointed consultants reported witnessing research fraud Geggie (2001) J Med Ethics 27:344–6
• On average 2% of scientists admit to have falsified research at least once and up to 34% admit other questionable research practices Fanelli (2009). PLoS ONE 4(5): e5738
• 31% of submissions to a Chinese journal were plagiarized. Zhang. (2010) Nature 467:153
• 30% of submitted manuscripts included plagiarism from a previous publication of the senior author. 16% of submitted manuscripts included plagiarism from another investigator’s work and/or website. McFarlin et al (2010) IJ Exercise Sci (3):3
• 9.5% of authors in the BMJ were ‘guest authors’: i.e. they did not make any significant contribution to the paper Bates et al (2004) JAMA. 292(1):86-8.
How dangerous is it?
• erodes public trust in the scientific community• is extremely costly to investigate and police• may create a flawed knowledge base (systematic
reviews)• may negatively influence funders and politicians• may divert funding and effort away from reputable
research• may influence the political process, esp in
controversial areas e.g. stem cell research (Hwang Woo Suk)
• can create false hope among patients
Corrections, Retractions and
"Expressions of Concern"
“The retraction or expression of concern, so labelled, should appear on a numbered page in a prominent section of the print journal as well as in the online version, be listed in the Table of Contents page, and include in its heading the title of the original article. The text of the retraction should explain why the article is being retracted and include a complete citation reference to that article.”
Uniform Guidelines for manuscripts submitted to biomedical journals (The Vancouver Guidelines) http://www.icmje.org
But retracted papers live on …
“Examination of one Nature paper by former Bell Labs physicist Jan Hendrik Schön, published in 2000 and retracted in 2003, revealed that it's been noted in research papers 17 times since, although the drop-off after retraction was steep: prior to being pulled, the paper was cited 153 times.”
Unger K. Couzin J. Even Retracted Papers Endure. Science 2006;312(5770):40-41
“The difference between citation levels of flawed originals and corrected republications does not approach statistical significance until eight to twelve years post-republication.”
Peterson GM. The effectiveness of the practice of correction and republication in the biomedical literature. J Med Libr Assoc. 2010; 98(2): 135–139.
Brice, J. and J. Bligh. Author misconduct: not just the editors' responsibility. Medical Education, 2005. 39(1): p. 83-89
Smith, R. The trouble with medical journals. London: RSM Books 2006
Hames, I. Peer review and manuscript management in scientific journals: Guidelines for good practice. 2007, London: John Wiley and Sons.
Wager, E., F. Godlee, and T. Jefferson. How to survive peer review. 2002, London: BMJ Books.
Bligh, J. and J. Brice, New Year reflections on professionalism. Medical Education, 2005. 39(1): p. 2-3.
Organisations:ICMJE: http://www.icmje.org/World Association of Medical Editors: http://www.wame.org/Committee on Publication Ethics: http://www.publicationethics.org/
Further information
