scary statistics: understanding risk for journalists

7/30/2019 Scary Statistics: Understanding Risk for Journalists

1/34

An adventure in the perception of news

Understanding Risk

Rebecca Goldin, Ph.D.Director of Research, STATS

Professor of Mathematics, GMU

January 17, 2013

National Press Foundation


2/34

Statistical Assessment Servicewww.stats.orgJon Entine, Senior Fellow

Cynthia Merrick, Intern

Rebecca Goldin, Director of

Research

Trevor Butterworth, Editor


3/34

Statistical Concepts in

Writing about RiskMean, median, mode

Standard deviation

Confidence intervals

Orders of magnitude

Confounding factors

Percentages

Absolute vs. relative risk

Scientific methods

Causation versus correlation


4/34

Absolute versus relative risk

Absolute risk is the risk you actually undergo.Women who take the birth control pill have an

absolute risk of venous thrombosis (blood clot) of

about 1 in 10,000 per year. The absolute risk of

women who do not take the pill is 1 in 15,000 peryear.

Relative risk is a risk compared to another group.

Women who take the birth control pill have a 50%

increased risk of venous thrombosis, compared to

women who dont take the pill.


5/34

Relative risk

representations

haveconsequence

In 1995, Committee on

Safety of Medicine in UKconcluded that the 3rd

generation birth control pill

was riskier than previous

versions.

Some press reported a

100 percent increase in

risk in Deep Vein

Thrombosis (blood clots);

others reported twice therisk.

The absolute riskfor DVT was

15 per 100,000 for 2nd

generation birth control pills

The absolute risk for DVT was30 per 100,000 for 3rd

generation birth control pills.

The media blitz led to many

women not taking theirmedications (rather than

immediately replacing them)

and an increase in

unwanted pregnanciesThe abortion rate went up 9%

from 1995 to 1996.

The absolute rate of DVT for

pregnant women is 80 per

100,000.


6/34

Kids dying from the flu

Detroit Free Press (Jan 16, 2013):Risk to allages: About 100 children die of flu each year

Story has many specific examples of children dying,with parents describing the tragedies

Prevalence of flu vaccine discussed (About 40% ofchildren are vaccinated)

Effectiveness of flu discussed (60% of people whowould have gotten it do not). Most deaths amongthose not vaccinated

24,000 per year die of all ages in the U.S.

No mention ofabsolute risk: there are about 74million children in the United States. The risk of deathby flu is 1 in 740,000 per year, or .00014 percent.


7/34

Comparisons are powerful for

risk

Cars Guns

~250,000,000registered cars in theUnited States(Department ofTransportation)

~32,000 car crashfatalities (NationalHighway Traffic Safety

Administration, 2010) 12.3 road fatalities per

100,000 people, 1.5-3times as many as inEurope (8.7 in Italy, 6.9

in France, 4.5 inGermany)

~130,000 federallylicensed firearms dealersin the United States

(Bureau of Alcohol,Tobacco, Firearms andExplosives); ~150,000gas stations, ~14,000McDonalds

270,000,000 guns incirculation (survey, 2007))

~30,000 people die eachyear including 8,500 bymurder (2011, FBI),19,000 by suicide (CDC,

2009), 1,000 by accident Rate of death by firearm


8/34

Causation or Correlation


9/34

Its easy to be fooled

Height correlates with reading skills in childrenunder 10.

Income correlates with success in college.

Ratio of finger lengths correlates with aggression. Facebook correlates with poor grades.

Facebook correlates with good grades.

Doing heroin correlates with doing marijuana.

Higher taxes correlate with high annual growth,and are inversely correlated with poverty rates.

Alcoholism correlates with less gray matter in the

prefrontal cortex.


10/34

IVF pregnancies may increase risk of blood

clots, blocked arteries Fox News, Jan 16, 2013

In vitro fertilization may come witha slight associated risk: blood clots

and blockages.

previous studies have found IVF

to be just as safe as normalpregnancies, but [the authors of

the new study] werent necessarily

convinced

Study Design:About 24,000

women who had undergone IVFwere compared to about 115,000

women who had normal

pregnancies. Each had average

age of 33.


11/34

Those who went IVF were more likely to haveblood clots. 4.2 out of 1000 had VenousThrombembolism among IVF women, comparedto 2.5 out of 1000 for other women.

.08 percent of IVF women had a blocked artery,while only.05 percent of women with normal pregnanciesdid.

But is it the IVF that increasedthe risk, or is IVF

reflective of an increase level of risk? Women ineach group do not have the same health profile.

Non-IVF contributors to these different numbers:different levels of fertility, access to medical careand diagnosis, hormonal treatments not directly

related to IVF IVF re nancies carr more risk of blood clots

IVF pregnancies may increase risk of blood

clots, blocked arteries Fox News, Jan 16, 2013


12/34

fMRI studies a case study fMRIs are large magnets

measuring oxygen levels inblood

People can engage in

activities inside the machine Patterns of blood flow are

thought to reflect patternsof brain activity (more on

that in a bit). Typical studies: assume that observed patterns

onlyoccur when the tested behavior occurs.

Typical studies: assume that observed patternsare causedby the tested behavior.


13/34

fMRI studies a case study Lying can be determined by

patterns of fMRI scans.

But perhaps stress or anxietycan lead to the same patterns

Violent video gaming leadsto violent brain patterns

But perhaps any competitiveplay, including non-violent

non-video games has similar brain patterns. Plus, noindication ofactualviolence.

Math anxiety triggers activity in the pain center of thebrain.

But no pain experienced by subject with math anxiety.Perha s anxiet , not mathematics, correlated with


14/34

Jumping from Correlation to

Cause You dont always have to know whyit may not be

causal. Be wary ofanyclaims of causality.

Some common reasons that a correlation could look

causal when its not include: not adjusting for

confounders, misunderstanding the mechanism,having an unknown confounder.

A causal relationship might be reasonable to suspect

when the statistics are

Overwhelming Observed in many different contexts

Repeated tests show the same effect, on large numbers

of people

Double blind case-control studies.


15/34

Causation vs. correlation is not the only

thing to worry about in medical research


16/34

The roll of randomness

Given a hug urn of balls 30% of the balls are white,and the rest are other colors.

Each of 100 people pick

10 balls, write down their

colors, then return the

balls to the urn.

Some people will have

3 white balls, but others

will have greater or fewer.


17/34

0

0.05

0.1

0.15

0.2

0.25

0.3

0 1 2 3 4 5 6 7 8 9 10

Probability

Number of White Balls Chosen

Randomness in Choosing 10 Balls:

How many are white?

Number of White Balls is

Random

Suppose that white represents something

random, and bad, like the number of cancers per100 people in a town.

If one town gets 7 cancers per 100 (double the

expected number), wouldnt you think theres a

reason? Our statistics suggest maybe not. But

About 27% chance

you will get 3 white

balls; its much more

likely youll get some

other number

About 1% chance of

getting 7 white balls


18/34

0

0.05

0.1

0.15

0.2

0.25

0.3

0 1 2 3 4 5 6 7 8 9 10

Probability

Number of White Balls Chosen

Randomness in Choosing 10 Balls:How many are white?

Randomness has structure

You can predict how likelythe data is, if you assumethe probability of white is 30%.


19/34

On p-values (how likely am I to see the

data I see, if the data are random?)

Suppose you are flipping a coinmany times, and you think this coin

is biased, because you arent

getting close to heads and

tails. How can you quantify yoursuspicion?


20/34

On p-values (how likely am I to see the

data I see, if the data are random?)

The p-valueis a measurement based on the data you haveseen: it answers the question: if the coin werefair, how likelywould I be to see the data I am seeing? In other words, ifyou had a fair coin, is it reasonable to see the proportion ofheads/tails, or is it very unlikely to see that?

If you flip 1000 times, and you get 520 heads, there is justunder a 10% chance of getting this many heads (or more). Incontrast, if you had 550 heads in 1000 flips, the chance ofthis happening randomly is only about

.1%., i.e. very unlikely if the coin were fair. The biomedical communit enerall acce ts =.05 5% as


21/34

Multiple Testing, or How to

Guarantee Results

Once you have a standard, like p


22/34

What happens in the lab:

Experiments Galore...


23/34

What the

rest of the

world sees


24/34

Metaphors for Bad Statistical

Methods

Drunk looking for his keys under the lamp post


25/34

Metaphors for Bad Statistical

Methods

Texas Sharpshooter Fallacy


26/34

Depression linked to increased

stroke mortality Heartwire

Based on data from NHANES I EpidemiologicFollow-up Study (National Health and Nutrition

Examination Survey)

But if you just comb the data, you are likelytofind something juicy. This doesnt mean itsuntrue.

We need to be savvier about challenging

scientists that take advantage of randomness togenerate spurious results

"Depression is not currently routinely screened

for after a patient has had a stroke," the author

of the study said in an interview. "We think it


27/34

Causation, Correlation and Risk

Correlation is not always clear

Causation is often inferred

Risks are often over-sold

A j li t ibl


28/34

Media Impact is

Great

In 1998, Andrew

Wakefield published astudy on 12 childrenwhich was the basis forthe belief that Autism isa result of vaccinations.

Press repeatedlyreported these results,even though thescientific community wasunable to reproduce theresults. The existence ofthis study gave greatervoice to other studies

A journalist was responsiblefor an investigation of thescientific integrity ofWakefields work.

After his autism study wasdiscredited, most mediacoverage about vaccinationsreports both sides of thestory about whether

vaccines are safe or not. However, the medical

community almostuniversally endorsesvaccine and believes that

vaccines are safe. Pockets of measles and

croup due to vaccinationrefusal or lack of herd effecthave been found in the U.S.and in the UK.


29/34

Basic Advice for a Journalist with

Limited Time and Ideas

Read as a skeptic at all times. Avoid most

conclusions of causality. A lot can be understood by even a cursory read (


30/34

Research is routinely plagued

Research is plaguedWhat can a journalistdo?

Low levels ofsignificance

Multiple testing

No acknowledgement ofrandomness in researchdesign

Lack ofcontext/repeatedexperiments

Scientists dont knowhow to talk to

journalists.

But if you are looking tofind one scientist willing

Write about the levels ofsignificance, bias,caveats

Ask the researchersabout multiple testing.Did they adjust forthem?

Write about absolute

risks. Look for a body of

research rather thanone specific paper

Cite your sources!

DONT INDICATE


31/34

Doting on Data

Lessons to be Learned

There is no certainty, due to random effects.

Dont over-estimate the ability of poor data to

give answers. Also, lots of data is unavailable.

Risks need to be contextualized.

Consensus is extremely important

The world is complicated; many things interact

with each other. The public voice is at leastas loud as the

scientific voice.


32/34

To Life!


33/34


34/34

Thank you!

scary statistics: understanding risk for journalists

Documents