DON’T BELIEVE EVERYTHING YOU READ: AN EDITOR’S VIEW

JAMES R. SCOTT, MD

I have no conflict of interest to disclose.

OBJECTIVES

• TO CLEARLY UNDERSTAND EVIDENCE BASED MEDICINE

• TO BE ABLE TO ASSESS VALIDITY OF NEW TREATMENTS

• TO APPLY BEST-EVIDENCE IN YOUR CLINICAL PRACTICE

JOHN IOANNIDIS

• WHY MOST PUBLISHED RESEARCH FINDINGS ARE FALSE. PloS Med 2005;2(8):124

• “The most downloaded document of all time on PubMed”

SCOTT’S RULE

IT WILL NEVER WORK AS WELL IN YOUR PRACTICE AS REPORTED IN THE LITERATURE

"BELIEVE NOTHING THAT YOU SEE IN THE NEWSPAPERS [TV, INTERNET].... IF YOU SEE ANYTHING IN THEM THAT YOU KNOW IS TRUE, BEGIN TO DOUBT IT AT ONCE."

Sir William Osler

EVIDENCE BASED MEDICINE: A RECENT PHENOMENON

RCTs

0

200

400

600

800

1000

1200

1400

1973 1983 1993 2003

Meta-Analyses

02000400060008000

10000120001400016000

1973 1983 1993 2003

2012 – 426,853 RCTs 2012 – 53,042 Meta-Analyses

NUMBER OF PUBLISHED PAPERS WITH “EVIDENCE BASED” MEDICINE IN TITLE

0

2000

4000

6000

8000

10000

12000

14000

1992 1996 2002 2004 2007 2009

2013 – NOW > 100,000

EVIDENCE-BASED MEDICINE

DE-EMPHASIZES• INTUITION• EXPERIENCE• PATHOPHYSIOLOGYEMPHASIZES• PROOF• OUTCOMES

PRINCIPLES OF EVIDENCE-BASED MEDICINE

• FIRST DO NO HARM• WHAT ARE THE RESULTS? • ARE THE RESULTS VALID?• HOW DO I APPLY THEM TO MY PATIENTS?

LEVEL OF EVIDENCE

I RANDOMIZED CONTROLLED TRIAL – The “Gold Standard”II-2 COHORT OR CASE CONTROL STUDYII-3 LARGE CASE SERIES III CASE REPORT, EXPERT COMMITTEE,

RESPECTED AUTHORITY, CLINICALEXPERIENCE

RULES TO IMPROVE PUBLISHED PAPERS

• LEVELS OF EVIDENCE• GUIDELINES – CONSORT, PRISMA, STROBE• TRIAL REGISTRATION• CROSSCHECK FOR PLAGIARISM• DECLARE ALL CONFLICT OF INTERESTS• DETECT FRAUD

PREFERRED STUDIES - RANDOMIZED CONTROLLED TRIALS (RCT) BEST

• MOST RELIABLE EVIDENCE • CONSORT GUIDELINES• DESIGNED FOR EFFICACY ONLY • MY RULE - INCLUDE ABSOLUTE RISK

– ACTUAL NUMBERS, PERCENTAGES– NUMBER NEEDED TO TREAT (NNT)

• NOTE – 25% LATER REFUTED

PROBLEMS TRANSLATING RCTs INTO PRACTICE

• DIFFERENT PATIENTS• COSTS MORE• INSURANCE WON’T COVER• PATIENT DOESN’T WANT IT • YOU ARE BETTER AT SOMETHING ELSE• UNANTICIPATED COMPLICATIONS OR

SIDE EFFECTS

CHALLENGES WITH NEW SURGICAL AND OBSTETRIC PROCEDURES

• RCTs DIFFICULT TO DO • BLINDING NOT POSSIBLE• COHORT/CASE SERIES NEXT BEST• UNETHICAL NOT TO COMPARE WITH CURRENTLY ACCEPTED METHODCAUTION: IN YOUR OWN PRACTICE

USE IRB & INFORMED CONSENT

CLINICAL JUDGEMENT STILL IMPORTANT

• WOMEN UNDERGOING ABDOMINAL SACROCOLPOPEXY RANDOMIZED TO BURCH OR NO BURCH (Brubaker et al. N Engl J Med 2006;354:1557)

• BURCH LOWERED INCIDENCE OF SUI FROM 44% TO 24% (20%)

• REAL WORLD – PT WITH VARICOSITIES BLEEDING, HEMATOMA & 8 UNITS OF BLOOD

IMPORTANT TO REMEMBER

EFFICACY – UNDER IDEAL CONDITIONSEFFECTIVENESS – IN REAL CLINICAL SETTING

(YOUR PRACTICE)

• NO RCT FITS EVERY PATIENT• ALMOST NO RCT FITS ANY PATIENT PERFECTLY• NOT GOOD AT DETECTING COMPLICATIONS

META-ANALYSIS: STATE-OF-THE-ART REVIEW

ADVANTAGES• STANDARD RULES • STATISTICAL POWERDISADVANTAGES• SUBJECTIVE ASSUMPTIONS• COMPLICATED, ARTIFICIAL• ONLY AS GOOD AS THE TRIALS USED • ODDS RATIOS & CONFIDENCE INTERVALS • TAKEN AS GOSPEL

COCHRANE REVIEWS

• MOST WELL DONE• PUBLICATION BIAS• LONG & BORING• SOME OUTDATED• LIMITED ACCESS• ODDS RATIOS ONLY

STATISTICS

A FELLOW WITH ONE LEG FROZEN IN ICE AND THE OTHER LEG IN BOILING WATER IS COMFORTABLE – ON AVERAGE.

J. M. Yancey

REALITY

NO RANDOMIZED TRIALS OR META-ANALYSES FOR ABOUT > 50-60% OF WHAT WE DO

NO RANDOMIZED CONTROLLED TRIALS

• VBAC• CESAREAN ON REQUEST• SOME GYN SURGICAL PROCEDURES• MANY NEW MATERIALS & DEVICES

EXAMPLE: THERE ARE NO RANDOMIZED TRIALS TO PROVE THAT PARACHUTES WORK

RECOMMENDATION:

EVIDENCE-BASED GURUSSHOULD PARTICIPATE IN A DOUBLE- BLIND RCT

Parachute Use to Prevent Death and Major Trauma. Smith GCS et al. BMJ 2003;327:149

COHORT STUDIES & CASE SERIES STILL USEFUL

• MORE CHANCE OF BIAS • STROBE GUIDELINES• HOW IT WORKS IN THE TRENCHES• LONG-TERM FOLLOWUP• FIRST REPORTS OF ADVERSE

EVENTS OR RARE COMPLICATIONS

LANDMARK OBSERVATIONAL STUDY

• FIRST DESCRIPTION OF AIDS BASED ON CLINICAL FINDINGS IN FIVE PATIENTS

• GOTTLIEB – 33 YR OLD ASSISTANT PROF.• SCHROFF – RESIDENT IN INTERNAL MED.

Gottlieb MS, Schroff et al. N Engl J Med 1981;305:1425

CITED 2,532 TIMES

SMALL CASE SERIES: OBSTETRIC COMPLICATIONS ASSOCIATED WITH THE LUPUS ANTICOAGULANT

D. WARE BRANCH MD JAMES R. SCOTT MD NEIL K. KOCHENOUR MD ED HERSHGOLD MD

N ENGL J MED 1985;313:1322

SIGNIFICANCE: Based on 8 Patients

• NEWLY RECOGNIZED SYNDROME• TREATABLE CAUSE OF FETAL DEATH• MAJOR ANTIPHOSPHOLIPID ANTIBODY

RESEARCH EFFORT NEXT DECADE

• CITED 583 TIMES

GOOD CASE REPORT

• FIRST SUCCESSFUL EMBOLIZATION FOR SEVERE POSTPARTUM BLEEDING*

* BY CHIEF RESIDENT @ U OF UTAH

BROWN BJ et al. Uncontrollable Postpartum Bleeding: A New Approach to Hemostasis Through Angiographic Embolization. Obstet Gynecol 1979;54:371.

• CITED 132 TIMES

WORTHY OF HEALTHY SKEPTICISM

• GUEST SPEAKERS • DATABASE STUDIES – Inaccurate• LARGE EPIDEMIOLOGIC STUDIES – Clinically

Irrelevant• DECISION ANALYSES – Soft data, assumptions• DRUG COMPANY SPONSORED – Ghostwriters

Bias, Overstated conclusions• ADVERTISEMENTS - Embarrassing

ALL TOO COMMON:

• WIDELY QUOTED IN MEDIA• SAME DATABASE AS WITH

MAGNETIC FIELDS• ? BIOLOGIC PLAUSIBILITY

NO INCREASED RISK IN EPIDEMIOL. SAME MO. (no publicity)

86 Papers – Conflicting Results

MY TAKE: ALL NONSENSECONCLUSION – COFFEE ASSOCIATED WITH MISCARRIAGE

WAKEFIELD 1998 PAPER IN LANCET RELATED MMR VACCINATIONS IN CHILDREN TO AUTISM

• DECLARED FRAUDULANT AND RETRACTED IN 2010.

• DID TREMENDOUS HARM • “SHOULD NEVER HAVE BEEN PUBLISHED”

Richard Smith, Former Editor of BMJ• SAME PATTERN OCCURING AGAIN MULTIPLE

WEAK ASSOCIATIONS --- PUBLICIZED BY MEDIA

• LARGE DATABASE• LOGESTIC REGRESSION

“MODELING”• Odds Ratios (95% CI) =

1.21 (1.01-1.46)• WIDELY PUBLICIZED BY

THE MEDIA

CONCLUSION: INDUCTION OF LABOR AND AUGMENTATION ASSOCIATED WITH AUTISM

INTERPRETIVE BIAS AND OVERSTATED CONCLUSIONS

• MISLEADING RESULTS - “3-FOLD INCREASED RISK …” REALLY 1/MILLION VS 3/MILLION - STATISTICALLY SIGNIFICANT BUT CLINICALLY IRRELEVANT

OTHER BORDERLINE “ASSOCIATIONS” WITH AUTISM DURING PAST YEAR

• Flu & Fever During Pregnancy• Obese Mothers, Weight gain • Short Pregnancy Interval• Maternal Thyroid Dysfunction• ICSI (IVF)• Older Fathers• Antidepressants, Gluten Sensitivity• F.H. of Autoimmune Disease• Lyme Disease• Air Pollution, Pesticides

TRUTH

ALL OF THESE EPIDEMIOLOGIC STUDIES WITH SMALL ODDS RATIOS SHOULD END WITH THE STATEMENT: “WE FOUND A WEAK ASSOCIATION AND WE HAVE NO CLUE WHAT IT MEANS”

David Grimes

CAUTION

• 75% CLINICAL TRIALS INDUSTRY FUNDED• PURPOSE - MAXIMIZE FINANCIAL RETURN• ONLY 1/3 NOW DONE IN UNIVERSITIES• WHO CONTROLLED DATA & WROTE PAPER• TIES WITH COMPANIES - MANY AUTHORS &

59% OF EXPERTS WRITING GUIDELINES • 5X > CHANCE OF FAVORABLE OUTCOME

WHEN COMMERCIALLY SPONSORED

PREDATORY OPEN ACCESS JOURNALS

• 8250 JOURNALS – 438 PUBLISHERS, 1/3rd IN INDIA

• ADVERTISE FOR PAPERS• CHARGE AUTHOR FEE TO PUBLISH• 304 VERSIONS OF FLAWED AND

FICTITIOUS WONDER DRUG PAPER SUBMITTED ACCEPTED BY 157 (>50%) SCIENCE 2013;342:60-65.

SUBTLE PROBLEMS DESPITE THE RULES AS INDUCTION AND MAINTENANCE THERAPY FOR ULCERATIVE COLITIS. NEJM 2013;369;699

BOTTOM LINE: MODERATE EFFECT AT 6 & 52 WEEKS

• Complicated RCT – 211 centers in 34 countries

• Compared drug vs placebo• 16 Authors – 10 with multiple ties to

industry – 7 Company Employees• Company held & analyzed data• No mention of probable cost

($100,000 per year)

• Company sponsored RCTs• Marketing in the guise of

research – to publicize expensive drugs

• Open Label, No Control Group, Short-term, Physician payment Example: VIOXX

WHAT NEEDS TO HAPPEN:

• INDEPENDENT INVESTIGATORS • COMPARE TO PRESENT DRUG/Rx• INCLUDE COST COMPARISON• REJECT EPIDEMIOLOGY STUDIES

WITH ODDS RATIOS < 3-4• REQUIRE ABSOLUTE NUMBERS OR NNT• TRANSPARENCY – DISCLOSE ALL

CONFLICTS INCLUDING EDITORS

• CLINICALLY RELEVANT

• VALID STUDY DESIGN

• STRINGENT REVIEW PROCESS

• RESULTS PHYSICIANS CAN TRUST

DON’T WASTE YOUR TIME - OTHER RELIABLE AND USEFUL SOURCES

• COCHRANE LIBRARYwww.cochrane.co.uk

• ACOG PRACTICE BULLETINSwww.greenjournal.org

• UP TO DATE www.uptodate.com• MEDICAL LETTER www.medicalletter.com

CHALLENGE

TO TRANSLATE IMPERSONAL & DOGMATIC STATISTICS INTO PERSONALIZED CARE OF REAL FLESH & BLOOD PEOPLE.

STILL IMPORTANT

• COMPASSION• EMPATHY• COMMUNICATIO

N• CLINICAL

JUDGEMENT• COMMON SENSE• ACCESS

REFERENCES

1. Scott JR. Show me the evidence. Obstet Gynecol 2002;100(3):403-4.2. Ioannidis JPA. Why most published research findings are false. PLoS

Med 2005;2(8)e124 3. Scott JR. Improving systematic reviews for clinicians: a journal

editor’s view. Paediat Perinat Epidemiol 2008;22(1):38-41.4. Scott JR. Evidence-based medicine under attack. Obstet Gynecol

2009;113(6):1202-3.5. Grimes DA, Schulz KF. False alarms and pseudo-epidemics. The

limitations of obsevational epidemiology. Obstet Gynecol 2012;120(4):920-7.

6. Smith R. The Trouble With Medical Journals. The Royal Society of Medicine Press Ltd. Edward Arnold Publishers. 2011, London, UK

7. Bohannon J. Who’s afraid of peer review? Science 2013;342:60-7.