Lessons from reviewing 1,889 stories Gary Schwitzer Publisher HealthNewsReview.org

Our criteria: Does the story explain Whats the total cost? How often do benefits occur? How often do harms occur? How strong is the evidence? Are there alternative choices? Is the condition exaggerated? Is this really a new approach? Is it available? Whos promoting this? Do they have a financial conflict of interest? 69% 66% 65% 61% 57% Percent unsatisfactory after 1,889 story reviews 7 years

Ignore or Minimize Potential Harms

Most common flaws Conveying a certainty that doesnt exist Exaggerating effect size Using causal language to describe observational studies Failing to explain limitations of surrogate markers/endpoints Single source stories with no independent perspective Failing to independently analyze quality of evidence

Bad science is no excuse for bad journalism.

Exaggerating effect size Absolute vs. Relative Risk Two ways of saying the same thing One way relative risk reduction makes effect size seem larger. Risk of something happening in one group relative to risk in another. Other way absolute risk reduction makes effect size seem smaller. The chance that something will happen to someone a ratio of number of people who have an event divided by all of the people who could have the event. We absolutely think you should use the absolute figures.

Tamoxifen cut breast cancer rate in half

Chance of breast cancer over 6 yrs. Women on placebo 3.3% Women on tamoxifen 1.7% 50% relative risk reduction 1.6% absolute risk reduction OR if 100 women took tamoxifen instead of placebo for 6 years, there would be 2 fewer cases of breast cancer.

Oh, by the way. Tamoxifen users were >2 times more likely to develop uterine cancer or blood clots in lungs/legs. In relative risk terms, thats 210% greater risk! (Absolute risk increased from 0.5% up to 1.1%) Net effect for every 1,000 women on tamoxifen: 16 fewer breast cancers 5 more serious blood clots 6 more uterine cancers

Risk reduction must be stated in absolute, not just relative, terms NBC News reported, In women, aspirin reduces the rate of stroke blood clots in the brain, the most common type, by 24% but had little effect on heart attack risk. In men, the daily aspirin cuts heart attack risk by 32 percent but had little effect on stroke risk. HealthNewsReview.org comment: 24 percent of what? And 32 percent of what? The journal article upon which the story is based did state the absolute values for both benefits and risks. Over 6.5 years: Daily aspirin is predicted to result in: - 2 fewer strokes and 2.5 more major bleeds per 1000 women - 8 fewer heart attacks and 3 more major bleeds per 1000 men. To many viewers, those numbers will mean more than 32% or 24%.

Last example: absolute vs. relative 3rd graf of NYT story: About 8 percent of 2- to 5-year-olds were obese in 2012, down from 14 percent in 2004. From the published study: Overall, there have been no significant changes in obesity prevalence in youth or adults between 2003-2004 and 2011-2012.

Results suggest a systematic bias of major newspapers preferentially covering medical research with weaker methodology. More likely to cover observational studies & less likely to cover randomized clinical trials When they cover observational studies, they select those of inferior quality.

Observational study vs. randomized clinical trial Observational study: observing what happens to people under exposure conditions that have been self-selected or determined by influences outside the control of the researcher. The researcher can choose what exposures to study, but does not influence them. (Susceptible to bias and confounding factors) Randomized clinical trial: A true experiment. The researcher controls the exposure and assigns people randomly to experimental or control group.

Using causal language to describe observational studies Observational studies cannot prove cause and effect so it is inaccurate to use terms like benefitsprotectsreduces risk These studies can only show a statistical association, so all you can say is that. We offer a primer, Does The Language Fit The Evidence? Association Versus Causation.

Association Causation

Stories on: Alcohol can make men smarter CBS Eating greater variety of fruit reduces risk of Type 2 diabetes NYT Small reduction in sodium in American diet could save half a million lives/yr. NYT Coffee may protect against heart failure ABC, CBS, HealthDay, CNN The St. Louis Post Dispatch reminded readers: Researchers caution, however, that they cant be sure whether these associations mean that drinking coffee actually makes people live longer. Coffee can kill you Atlanta magazine Coffee may cut risk for some cancers CNN Reader comments: I am so fing sick of these studies. Stop wasting our time. Correlation IS NOT causation!!!! The statistics book in a class Im taking uses coffee as an example of statistics run amok.

Surrogate markers/endpoints Measuring an outcome, such as a lab test, that substitutes for measuring an important life event but may not be an important finding in itself. A difference to be a difference must make a difference Drug may reduce cholesterol but not prevent MI/death Drug may increase bone density but not reduce hip fractures Cardiac Arrhythima Suppression Trial (CAST) drugs to reduce premature ventricular contractions. Study stopped early because drugs led to higher death rate and nonfatal problems than placebo. Avastin for breast cancer - drug may extend progression-free survival but not impact overall

Failure to explain limits of surrogate markers USA Today: New drug may turn back the clock on heart disease. HealthNewsReview.org analysis: The story focuses excessively on the unprecedented changes in cholesterol achieved with the drug. these are surrogate markers that may not reflect any real benefit on the outcomes that matter to patients. WebMD: Beet Juice Good for Brain: Drinking Beet Juice Increases Blood Flow to Brain and May Fight Dementia. HealthNewsReview.org analysis: Surrogate endpoints do not necessarily reflect on overall health outcomes. And especially not from a study of 14 people over four days. And from this we get the headline that drinking beet juice may fight dementia ??? HealthNewsReview.org linked to 10 stories that framed a tiny short- term study of a biochemical marker of bone turnover as if it were definitive advice to women that, for example, Two glasses of wine a day ends menopause misery.

Whom to believe? Framing matters. Dueling news stories on new prostate cancer radiation therapy: Breakthrough? Or not a home run? Medical News Today: breakthroughextends lives vs. NY Times Well blog (quoting independent expert): I think this is a big deal. Its not a home run, but its a nice advance.

Whom to believe? Framing matters. Dueling news stories on brain wave scan for ADHD. Important tool? Or waste of money? NJ Star-Ledger, Fox News, NY Daily News, Bloomberg all gushing praise such as this should be an important tool quote from MD in the study vs. NY Times quoted independent expert skeptical about the test LA Times quoted independent expert The current scientific research really doesnt support (this) as a diagnostic tool. I would caution people. ABC News quoting independent expert: I dont know that this going to help...its going to make people spend money needlessly. They can charge for it and it gives you a pseudo-scientific basis for diagnosis.

Whom to believe? Framing matters. Dueling news stories on melanoma detection device. Helping doctors? Or causing dissent/polarizing the field? Cleveland Plain Dealer: helps doctors detect deadly cancer WTVF Nashville & KFSN Fresno among the stations using a syndicated news services story that claimed the Cleveland Clinic named the device one of its top medical innovations of 2013. KDVR Denver allowed a doctor to call it the biggest advance in melanoma in his 26-year career vs. NY Times quoted 3 independent experts: 1. This should still be considered to be in the developmental stage. 2. A dermatologist on the FDA panel who voted against approval 3. Biostatistician: (The test) just says everything is positive. I dont think it helps an aggressive doctor and unaggressive doctors could do just as well if they were more diligent without the device.

Whom to believe? Framing matters. Dueling news stories on meds for mild-moderate depression. Give pause about prescribing? Or no reason to avoid meds? HealthDay: Severely depressed gain most from antidepressants: But thats no reason to avoid meds for moderate cases. vs. LA Times: Study finds medication of little help to patients with mild, moderate depression USA Today (quoting independent expert): Such findings demonstrate a failure in the system: These drugs were not thoroughly tested in mild to moderate depression prior to their approval.

Whom to believe? Framing matters. Dueling news stories on Alzheimers drug. BY THE SAME NEWS SOURCE! Reverses disease in mice? Or a flop? MedicalDaily.com: Cancer drug may help reverse Alzheimers. AND Targretin a flop: Potential Alzheimers drug fails retrial.

Whom to believe? Framing matters. Dueling news stories on limitations of colonoscopy. Miss many cancers? Or concerns overblown? NY Times: Colonoscopies miss many cancers, study finds. VS. ABC News: The Case for Keeping Colonoscopy: Cancer Experts Say Concerns Over Colonoscopy Effectiveness May Be Overblown. CBS News Katie Couric: Dont use this study as an excuse not to get screened.

Whom to believe? Framing matters. Dueling news stories on prostate cancer screening study. Reduces deaths? Or isnt saving lives/value questioned? MSNBC: Regular prostate screening reduces deaths HealthDay: PSA testing cuts cancer death risk Bloomberg: PSA screening reduces deaths vs. Associated Press: prostate cancer screening isnt saving lives, study finds CNN.com: Value of mass prostate cancer screenings questioned.

The final line of the NEJM article: More information on the balance of benefits and adverse effects, as well as the cost-effectiveness, of prostate- cancer screening is needed before general recommendations can be made. Maybe that should have received a little more attention.

Framing Star Tribune: Boy, 12, dies after historic transplant to treat HIV, leukemia. He was in line to become the second person in the world to be cured of both deadly illnesses by the extraordinary type of bone marrow transplant, doctors said. Maybe. But in fact he was in line to die from a transplant that was not historic because it did not treat his HIV & leukemia.

The tyranny of the anecdote Stories may use only positive, glowing patient anecdotes but fail to capture trial dropouts, compliance problems, patient dissatisfaction or the choice to pursue less aggressive options. In one example, four major US news sources all used the same breakthrough patient anecdote of one mans reported improvement from a heart failure stem cell experiment. This suggests another common problem: public relations people spoon-feed journalists the patient anecdotes that put an intervention in the most positive light.

Single source stories & journalism via news release are unacceptable ABCs Good Morning America asked, Could a cure for obesity be just a pill away? HealthNewsReview.org analyzed the sourcing in this story: In a bizarre twist, the reporter used the potential conflicts of interest of the doctor he interviewed in the studio as evidence of his expertise. He noted that the doctor is a consultant to the manufacturer of the pill and then said, Youre the right man to talk to. Really? In a recent journal publication, this researcher listed financial ties to 17 drug and device makers, including the maker of the (pill in question) viewers were not clearly told that all of the results reported in the story came from a trial controlled by the company that has not been independently reviewed. The Wall Street Journal reported, Depression treatment Cymbalta effective as pain reliever, Lilly says. HealthNewsReview.org wrote: This 189-word story failed on almost every count: no independent perspective; a quote apparently lifted from a company news release; The company news release actually did a better job of explaining the study than did this news story.

Stories about screening tests Sir Muir Gray: All screening programmes do harm; some do good as well. All screening stories should explain the tradeoffs of potential benefits and potential harms

Rates of new diagnosis and death for five types of cancer in the US, 1975-2005. Moynihan R et al. BMJ 2012;344:bmj.e3502 2012 by British Medical Journal Publishing Group

Unquestioning news coverage of disease awareness campaigns

Movember misinformation in Chicago Sun-Times from star of In the Bedroom with Dr. Laura Berman She wrote that men arent aware of the warning signs of prostate cancer and went on to list some, including frequent lower back pain and swelling of the legs and feet and weight gain. But there are no warning signs that are specific for prostate cancer. Her message could create unnecessary anxiety and lead to unnecessary testing. She mentioned an industry-funded survey suggesting that men greatly underestimate their risk of prostate cancer. She did not provide a reference for that survey. But peer-reviewed published work has shown that men vastly overestimated such risk. She also wrote as a statement of fact that more frequent ejaculation could help men to lower their risk of prostate cancer. At best, the research on this point is inconclusive, with other studies suggesting that men who have more ejaculations are more likely to develop prostate cancer. She may know something about ejaculation but shes shooting blanks on this one.

Fawning coverage of new technologies (proton beam therapy & robotic surgery) KERA public radio in Dallas reported on both technologies in its Battlefield Breakthrough series. Headline: Zapping Cancer With A Ray-Gun. We wrote: The story presents not one word of caveats, concerns or limitations. There was no independent source, no data about efficacy. Instead, there was fascination: looks like something out of Star Wars. Headline: Robots Slice Time Under The Knife. We wrote: Star Wars was a theme in that one as well. In fact, the formula was the same as for part one with proton more battlefield breakthroughs. Maybe we need to wait a bit to see how this arms race plays out with evidence and data before proclaiming battlefield breakthroughs just yet.

Going easy on health business stories Reuters Headline: Medtronic tests stents for erectile dysfunction. Our comment: An initial feasibility study in 30 men is worthy of news coverage? With a company-sponsored investigator as the only source? Not in our view. Chicago Tribune Headline: Area firms pneumonia treatment passes test. Our comment: It features the company presidents positive assessment of the study results but does not seek independent comment. It fails to make clear that the results of the trial have not been peer-reviewed or published and that results of a related trial have not been released at all. 56 New York Times Headline: Test of Eye Drug Is Said To Show Success in Elderly. Our comment: The piece shouldve slapped a bigger warning label on this cocktail of leaked summaries of secret results, speculation, and an inadequate dash of independent evaluation. Wall Street Journal Headline: AstraZeneca Says Brilinta Beats Plavix in Clinical Trial. Our comment: lets a drug company get away with making superiority claims without releasing data. All information in the story appears to come from company sources. Los Angeles Times Headline: Drug for menstrual cramps in the works. Our comment: This story reports that data were presented at a scientific meeting. But it didnt tell readers what the data showed. But it did allow a company VP the only person quoted to say this could be a breakthrough. Wow. Minneapolis Star Tribune HealthNewsReview.org found four stories within six weeks in which the Star Tribune established a clear pattern of writing nice things about local medical device industries, but did not lead in reporting negative stories (e.g., on troubling Medtronic stories in their own back yard, often scooped by Milwaukee Journal-Sentinel in another state, a smaller paper.)

Blog roll: Dr. Richard Lehmans weekly journal review on a BMJ blog. Funny. Hard-nosed. A treasure. Retraction Watch/Embargo Watch. Knight Science Journalism Tracker Skeptical Scalpel blog. Not Running A Hospital blog.

1. Not all studies are equal. All have flaws. You and your audience should appreciate the limitations inherent in any study design. 2. If you rely on medical journals as a main source of your news, you are getting and giving an imbalanced view of new treatments, tests, products and procedures. 3. Both industry interests and journal policies tend to promote positive findings. 4. If you rely on presentations at scientific meetings as a main source of your news, you may be promoting claims that have not been validated by any independent expert sources. 5. There are a growing number of resources that can help you evaluate claims and evaluate the underlying science in journal articles, presentations at scientific meetings, news releases and interviews with experts.

Good journalism has a subtle feature of reticence. We dont publish everything we hear. We filter. We curateand this requires the willingness to not publish things that are unlikely to be true. Joel Achenbach, Washington Post The Shroud of Turin, pseudoscience and journalism

8th straight AHCJ workshop discussing HealthNewsReview.org could be our last

Whos going to continue this work?

@Thank you gary@healthnewsreview.org