![Page 1: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/1.jpg)
Published research - flawed, misleading, nefarious
John HoeyUniversity of Waterloo, 2009
www.slideshare.net/hoeyj
![Page 2: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/2.jpg)
Why is research important - and what is it?
Can we believe what we read in journals?
What can be done?
![Page 3: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/3.jpg)
Why is research important - and what is it?
Public health practice Clinical practicePublic policy
RationalEvidence-based i.e. Research
![Page 4: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/4.jpg)
Why is research important - and what is it?
++ research designsRCTsHealth economicsGenomics ...
So far ~ 150 guidelines
![Page 5: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/5.jpg)
2,000 - 3,000 manuscripts /yrLancet - 6,000 to 7,000 /yr
![Page 6: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/6.jpg)
![Page 7: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/7.jpg)
Maintaining the Integrity of the Scientific Record.......why?
•Distorts the scientific corpus
•misleads other investigtors
•Harm to patients and the public
•Physical harm
•Financial harm
![Page 8: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/8.jpg)
It’s also ethically wrong
Nuremberg - Helsinki
“Both authors and publishers have ethical obligations.
In publication of the results of research, the investigators are obliged to preserve the accuracy of the results.
Negative as well as positive results should be published or otherwise publicly available.
Sources of funding, institutional affiliations and any possible conflicts of interest should be declared in the publication.
Reports of experimentation not in accordance with the principles laid down in this Declaration should not be accepted for publication.”
![Page 9: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/9.jpg)
Let’s look at an example
Qualitative EconomicRandomized Clinical Trials Systematic reviews/meta-analysesetc..
![Page 10: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/10.jpg)
RCT
RRx A
Placebo
Outcome
![Page 11: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/11.jpg)
Consort Statement•Poor quality of reporting RCTs.
•Could not be trusted for use in clinical practice
•Could not be combined in systematic reviews
•Identified 22 essential items needed to conduct a RCT.
Ann Intern Med. 2001;134:663-694. www.annals.org David Moher
![Page 12: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/12.jpg)
Research Reporting Guidelines
- Clinical Trials [Consort]•22 essential items/elements
•e.g Randomization
•Item # 8 How was the sequence generated?
•Item # 9 Allocation concealment
•Item # 10 Implementation
![Page 13: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/13.jpg)
Does this really matter?
![Page 14: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/14.jpg)
Miller AB, et al. Canadian National Breast Screening Study: 1. Breast cancer detection and death rates among women aged 40 to 49 years. CMAJ. 1992; 147(10): 1459–1476
Canadian trial of mammogrphy for breast cancer
Begun in about 198050,000 asymptomatic womenRandomly assigned to
mammographyusual care
Followed for 8 years
![Page 15: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/15.jpg)
•Item # 9 Allocation concealment
•Item # 10 Implementation
![Page 16: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/16.jpg)
Randomization in the Canadian National Breast Screening Study: a review for evidence of subversion. Bailar JC, MacMahon B. CMAJ 1997;156:193-9
Harms?
Can science be trusted?Ethical responsibility to study subjects
Today - still unclear if mammography is beneficial - harms to women, costs to society
![Page 17: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/17.jpg)
Schulz et al - Quality of RCTS
Cochrane Pregancy and childbirth data base
33 meta-analyses - 250 trials
Quality of randomization and risk estimates
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methogological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408-12 (currently cited 1777 times in )
![Page 18: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/18.jpg)
•Randomization
•Concealment of treatment allocation schedule
•Generation of allocation sequences
•Inclusion of all trial subjects in analysis
•Double blinding (subject and investigator)
Of 22 items - one example
![Page 19: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/19.jpg)
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methogological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408-12
Quality of 250 trials
Concealment
% of trials
![Page 20: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/20.jpg)
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methogological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408-12
Quality of 250 trials
% Exaggeration of Risk Ratio
% of trials
![Page 21: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/21.jpg)
Study design - RCT
RRx A
Placebo
Outcome
Primary Outcome -Specified in Protocol?or fishing expedition?
QuickTime™ and aPhoto - JPEG decompressor
are needed to see this picture.
![Page 22: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/22.jpg)
Selective publication - Outcome biaspublishing the more interesting (usually
positive) result
Was there an hypothesis? A plan for analysis and reporting
of data?
In an RCT, this is the primary outcome
![Page 23: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/23.jpg)
Selective publication - Outcome bias(publishing the more interesting result)
48 trials1402 outcomes
31% - 59% incompletely reported(40% not reported at
all)
Chan, A.-W. et al. CMAJ 2004;171:735-740
![Page 24: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/24.jpg)
Selective publication - Outcome bias(publishing the more interesting result)
Interpretation: Intensive multitherapy for patients with poorly controlled type 2 diabetes is successful in helping patients meet most of the goals set by a national diabetes association.
However, 6 months after intensive therapy stopped and patients returned
to usual care the benefits had vanished,
However, 6 months after intensive therapy stopped and patients returned to usual care the benefits had vanished.
![Page 25: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/25.jpg)
![Page 26: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/26.jpg)
Kay Dickinson, Reporting and other biases in studies of Neurontin for migraine, psychiatric/bipolar disorders, nociceptive pain, and neuropathic pain.August, 2008
http://dida.library.ucsf.edu/pdf/oxx18r10
Reporting Bias
![Page 27: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/27.jpg)
Some definitions • Positive= Study shows results favourable
to the intersts of the author/sponsor. Usually statistically significant.
•Negative = Study shows results unfavourable to the interest of the the author/sponsor - may or may not be statisticaly significant.
![Page 28: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/28.jpg)
Reporting Biases
Nefarious or Incompetent?•Non-publication of negative or
neutral results
•Selective publication of results - outcome bias
• Multiple publication bias
•Language bias - and publishing in the grey literature
•Time lag bias
•Undeclared conflicts of interest
•Ghost writing Dickinson
![Page 29: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/29.jpg)
images from Wikipedia which also has a nice summary of court proceedings and results
Neurontin(gabapentin)
![Page 30: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/30.jpg)
P Wessely, C Baumgartner, D Klinger, J Kreczi, N … - Cephalalgia, 1987
Bias Example
Publication Final negative primary results not published, only positive preliminary results
Selective outcome reporting
Outcome reported was not primary or secondary outcome
Selective statistical analyses
2 nonrandomized patients assigned to neurotin were include with those randomized
Spin Emphasis on “positive” outcomes
Neurontin for migraine headaches study
![Page 31: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/31.jpg)
16 Citations P Wessely, C Baumgartner, D Klinger, J Kreczi, N … - Cephalalgia, 1987
Who cares? General Principles of Migraine Management: The Changing Role of PreventionE Loder, D Biondi - Headache: The Journal of Head and Face Pain, 2005 - Blackwell Synergy
Preventive treatment of migraine - SD Silberstein - Trends in Pharmacological Sciences, 2006 - Elsevier
Migraine preventionDW Dodick, SD Silberstein - British Medical Journal, 2007 - pn.bmj.com
Neuromodulators for Migraine PreventionR Kaniecki - Headache: The Journal of Head and Face Pain, 2008 - Blackwell Synergy
Practice parameter: Evidence-based guidelines for migraine headache (an evidence-based review)
Stephen D. Silberstein, MD, FACP, for the US Headache Consortium
Neurology 2000;55:754-762
![Page 32: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/32.jpg)
![Page 33: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/33.jpg)
![Page 34: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/34.jpg)
www.flickr.com/photos/dariustwin/3362416244/
the story of Dr. Reckless and the pharmaceutical company - Pfizer
![Page 35: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/35.jpg)
“ Ghost authorship exists when someone has made substantial
contributions to writing amanuscript and this role is not
mentioned in the manuscript itself. WAME considers
ghost authorship dishonest and unacceptable”
www.wame.org/resources/policies
![Page 36: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/36.jpg)
Ultimately not published as a stand alone paper, but incorporated in a paper with other studies.
UK doc. (Dr. Reckless) tries to publish his research
Neuroptin for pain relief59 sites in 6 countries.
Study ends in 1999.
Results negative (no benefit)
By 2002, no manuscript produced.
Dr. Reckless writes to Pfizer
![Page 37: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/37.jpg)
Ghost Writing - the case of Dr. Reckless vs. Pfizer
“I don’t think we should be too hasty with this request,”
• “I agree with your answer. Although I would love to publish SOMETHING about 945- 224,
Donna McVey made it very clear that we should take care not to publish anything that damages neurontin’s marketing success.”
“..We would need to have ‘editorial’ control, but would certainly involve Dr. Reckless in the process, asking for his expert comment.”“
I think that we can limit the potential downsides of the 224 study by delaying the publication for as long as possible and also from where it is published.
More importantly it will be more important to how WE write up the study. We are using a medical agency to put the paper together which we will show to Dr. Reckless.
We are not allowing him to write it up himself.”
![Page 38: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/38.jpg)
So what?
http://dida.library.ucsf.edu/pdf/oxx18r10
In total, I find that there were 43 million off-label prescriptions of Neurontin as a result of the ... promotional activities related to the off-label uses ... that would not have occurred absent the challenged conduct.”
Meredith Rosenthal
![Page 39: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/39.jpg)
what can be done?
Education in researchSkepticism ++
*Conflict of interest*Use Reporting guidelines
![Page 40: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/40.jpg)
Conflict of Interest
![Page 41: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/41.jpg)
1109 Canadian clinicaltrials.gov732 investigators11 ideal practices to mitigate fCOI
- control over design- data collection- analysis- interpretation-write up-authorship-decision to publish-etc.
6% of investigators met all ideal practices
Rochon P, et al. - in peer review
![Page 42: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/42.jpg)
![Page 43: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/43.jpg)
Reporting GuidelinesConsort - RCTs
Prism- Systematic reviewsMoose - Meta-analysesStard - Diagnositic testsGrade - Clinical guidelinesStrobe - Observational studies
etc. (~160) www.equator-network.org
![Page 44: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/44.jpg)
Reporting Guidelines
Protocol Ethics Funding Write up Journal
![Page 45: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/45.jpg)
to sum up•Patient Care, Public Health & Policy •Rationale and Evidence Based
•Much research is•Unreliable, confused, distorted, conflicted
•So read cautiously and skeptically
•Be familiar with good research standards
•Use guidelines
•Teach others about them
![Page 46: Published Research, Flawed, Misleading, Nefarious - Use of Reporting Guidelines. University of Waterloo, 2009](https://reader035.vdocuments.us/reader035/viewer/2022070313/554b3356b4c905ce088b5102/html5/thumbnails/46.jpg)
Thank-you