Understanding Lies, Damn Lies, and Statistics: A Look At Why So

Many People Find Statistics Frustrating

John P. Holcomb, Jr.

Cleveland State University

Ohio MAA Section Meeting

April 1, 2005

Outline• Why do statisticians find public reporting of statistics frustrating?

• Why does the public find statistics frustrating?

• Why do students find statistics frustrating?

• What are some major differences between statisticians and mathematicians?

• Emphasize our similarities

"There are Three Kinds of Lies: Lies, Damn Lies and Statistics."

• Attributed to Benjamin Disraeli (1804 - 1880)

• Prime Minister (1868, 1874 -1880)

• Said to be popularized by Mark Twain in the United States

Statistics Affirming Quotations

• Frederick Mosteller (Harvard University)

• “It is easy to lie with statistics, but it is easier to lie without them.”

What Drives Statisticians Nuts?

Yahoo! News, (September 7, 2004)

Study Links TV to Teen Sexual Activity

• “Teenagers who watch a lot of television with sexual content are twice as likely to engage in intercourse than those who watch few such programs.” (Reuters)

• Rebecca Collins, “This is the strongest evidence yet that the sexual content of television programs encourages adolescents to initiate sexual intercourse and other sexual activities.”

• The problem is this is an Observational Study

• Did not sit 1,792 adolescents down and force them to watch television

• Adolescents chose their own “treatment”

Confounding

• Occurs when some other variable(s) affects both the independent variable (TV watching) and the dependent variable (Sexual Activity)

• Can be obvious and not-so-obvious

• This is hard for statistics students when it is covered in class, but for the public …

Problem with All Observational Studies

• Cannot assume there is no confounding

• So critics always have opportunity to criticize observational studies

• This is the defense of the Tobacco Industry for smoking

So why am I concerned?

• There is no mention of the role of parental supervision

• What is the consequence?

• The public misguided on the meaning of the result

Experiments

• Allow researchers to make “causal” conclusions

• Randomly assign subjects to “treatments” and “control” to ensure balance– Control does not necessarily mean “sugar pill”

• Both groups alike to every known variable as well as every unknown variable EXCEPT the treatment variable

Example II

• July 9, 2002, The Journal of the American Medical Association releases the results of the “Women’s Health Initiative (WHI)”

• Headlines Across America warned women about the risks from Hormone Replacement Therapy (HRT)

• New York Times: Study Is Halted Over Rise Seen In Cancer Risk

Belief: Estrogen and Progesterone would help women live healthier lives

Findings:

• Increased risk for breast cancer (26%)

• Increased risk of heart disease (29%)

• Increased risk of Stroke (41%)

• 1962 – Observational study suggests estrogen therapy reduces risk of breast and genital cancers

• 1980 – A study shows that estrogen and progesterone together reduce risk for endometrial cancer

• 1985 – The Nurses’ Health Study, with 121,964 subjects finds lower rate of heart disease in those taking progesterone

• 1995 – Same study finds that estrogen and progesterone reduce heart attack risk by 39%

Previous Good News

Ethical Question

• For the WHI can we deprive the control group this great treatment?

What Went Wrong?

• One major issue – Nurses’ Health Study is observational

• WHI is a clinical Trial• One theory is the confounder is health –

healthier nurses took the HRT and stayed on the HRT

• Another theory is the nature of the study – those who had some kind of heart ailment stopped taking medicine

• Even though WHI was a clinical trial (experiment), informed consent can add bias

• Also, Women in WHI were older (most were 60 or older instead of going through menopause)

Caution

• Observational Studies are not useless

• Often point to issues needing further investigation

– Experiments

– Animal Studies

What Did Not Make the Headlines (or Even the Article)

• Recall the earlier increase:– Breast cancer (26%) – 8 more cases for every 10,000 women– For 8 to equal 26% increase then:

81.26 8 .26 30.77

XX X

X

P(Breast Cancer in Placebo Group) = 31/10,000 = .0031

P(Breast Cancer in the HRT Group) = 39/10,000 = .0038

THESE ARE STILL VERY SMALL PROBABILITIES!

Frustrations:

1. Difference between observational studies and experiments is subtle

2. For statisticians, there is no contradiction, but for the public and even scientists, there is a glaring contradiction

3. Confirms the culture of disbelief – and who is blamed?

4. There is inherent uncertainty in the process

Statistics is Perfect for the Law

• Since all conclusions are based on probability – we can never say anything definitively

• 0 and 1 are difficult to achieve ever in practice

Implications for Teaching

• These are the topics we need to discuss– Study Design– Confounding and Causation– Treatment vs. Placebo– Absolute and Relative Risk– Uncertainty

• “All models are wrong, but some are useful” – George Box (University of Wisconsin)

Further Implications

• In the courses:– Introductory statistics– Statistical literacy– Mathematics for liberal arts

• Statistical thinking will one day be as necessary a qualification for efficient citizenship as the ability to read and write.– H.G. Wells

Rational vs Emotional

• Statistics and Mathematics have the perception of being rule enforcers

• People do not like being told what to do or what not to do

• We are constantly saying do not play the Lottery– My life is a personal failure

Mega Millions

• July 2, 2004

• Mega-Millions jackpot reaches $290,000,000

• Probability of winning is .000000007399 = 7.399x10-9

Fox News Cleveland

Dr. Killjoy

• 57 times more likely to die from a motor vehicle accident that day then win MegaMillions

• 21 times more likely to die from lightening strike in a year than win MegaMillions

Why Do Students Find Statistics Frustrating?

1. Stilted Language

– Recall an earlier phrase

• “Cannot assume there is no confounding”

– We are the masters of the double negative

Confidence Intervals

• Students want to say– The probability the mean is in the interval is

95%

• What we require them to say– “We are 95% confident the interval (a,b)

captures the unknown population mean”– When drawing random samples from a

population, calculating the intervals in this manner captures the unknown mean 95% of the time.

Hypothesis Testing

Want to say “Accept Null”

• Have to say “Fail to Reject Null”– (AND we make them put in context)

• Again we statisticians can’t be certain (or accepting) of anything

2. Look At What We Make Them Do

3. Statistics Taught By Folks Who Are Not Trained Statisticians

• Statistics was added “on the side” to their training

• Not sure of the “why”, so it is difficult to motivate

• Teaching statistics is “scraping the bottom of the barrel” in classroom assignments

• “In God We Trust, All Others Bring Data”

• W. Edwards Demming (TQM Guru)

• At CSU, there are at least 7 different departments teaching some kind of introductory statistics comprising over 100 faculty

• Only 4 faculty on campus have a Ph.D. in Statistics

• At many schools that may be even lower

Differences Between Mathematics and Statistics

• Statistics is too dirty

• Mathematics is pure and pristine

• Mathematics is built on axioms, definitions, and theorems

• Statistics is built on “flawed” processes right from the very beginning

Inferential Statistics

Giant Leaps of Faith

• Assume the population is definable

• Assume the population is stable

• Assume the sample is representative (bias free)

• If all this is true, then can we rely on Mathematics for our confidence interval to capture the mean 95% of the time.

• Often mathematicians want “perfect” studies or nothing

• “If you do not know what to measure, measure anyway, you’ll learn what to measure next time.”– David Moore (Purdue University)

• Assessment

X

No Quod Erat Demonstrandum

• I get a representative sample• The sample size is large enough to invoke

the Central Limit Theorem

• I calculate

• I still do not know if my interval contains the unknown mean

1.96s

Xn

ERGO

• I have to wonder . . .

• Mathematicians do not like uncertainty

Difference #2

• Applied Statisticians have to communicate with other researchers

• These researchers often have limited statistical training

• (Present company excluded), mathematicians are not exactly known for their patience with those deemed less worthy

• The main challenge is to take a scientific hypothesis and turn into a testable statistical hypothesis

• Have to convince researchers that input prior to collecting data is critical

– Cleveland Cavaliers

• Have to educate them not to “Stone the Messenger”

Difference #3

• Statisticians make more money

• Statisticians have more job options

• Go to icrunchdata.com

1-50 of 119 | First | Previous | Next | Last

J ob No. J ob Title Company Name Date Posted State Exp. Salary

825 Senior Marketing Analyst Advanced Financial Services, Inc.

3/28/2005 RI 5-8 80-89K

824 Employment Systems Analyst & Researcher

University of Connecticut 3/28/2005 CT 0-2 --

823 Senior Research Analyst - Fortune 100 Company

UnitedHealth Group 3/25/2005 MN 3-4 --

822 Manager, Statistical Analysis The Brixton Group, Inc. 3/24/2005 VA 5-8 100-109K, 110-119K, 120-129K, 130-139K, 140-149K

821 Sr. Statistician The Brixton Group, Inc. 3/24/2005 VA 5-8 90-99K, 100-109K

820 Business Analyst The Brixton Group, Inc. 3/24/2005 VA 0-2 50-59K, 60-69K

819 Informatics Statistics Manager, Senior/ Lead Informatics Analyst and Informatics Analyst

BSA Advertising for Aetna 3/22/2005 PA 0-2 --

818 DATABASE MARKETING SPECIALIST

Home Shopping Network 3/21/2005 FL 0-2 --

810 Statistician (Marketing) Vistrio 3/21/2005 OH 3-4 open

• Try going www.idoproofs.com

• Great Opportunities in Math

– 101 Careers in Mathematics

– http://www.maa.org

• BS in Mathematics

• MS in Mathematics

• Took Prelims in Real Analysis, Topology, Complex Analysis, and Math Stat

• Would have gotten a Ph.D. in mathematics …

• I do love Mathematics and Mathematicians

• HONEST!

My Own History

Why Can’t We Be Friends???

• Undergraduate Math Departments Need Math Majors

• Graduate Statistics Departments Need applicants

• We need to offer mathematically talented students as many options as possible

Easier Said Than Done

• We need to let undergraduates know what statistics is

• Traditional Probability and Statistics sequence is NOT statistics

• Students need authentic experience working with data

Enrollment

• 264,000 students took Elementary Statistics according the 2000 CBMS

• www.ams.org/cbms

• 77,000 to take AP STATS in 2005

• These people are NOT welcome in Mathematics Departments

If I were King of the World …

• Calculus I, II, III• Linear Algebra• Intro Proof/Discrete• Differential Equations

• Real Analysis• Probability• Math Stat• Applied Stats

Do Not Reinvent The Wheel

• The American Statistical Association has guidelines:– Majors– Concentrations– Minors– Google Search USEI Guidelines– Journal of Statistics Education

• www.amstat.org/jse

Shameless Plug …

• Check out an innovative statistics course for majors at www.rossmanchance.com (click ISCAT link)

• Beth Chance and Allan Rossman

• Investigating Statistical Concepts, Applications, and Methods (Duxbury)

• MAA PREP Workshop July 18-22

• www.maa.org/prep/2005

Goals

• Show specific examples of frustrating news stories involving statistics

• Discuss the importance of these “soft” ideas in low – level courses

• “Feel the Pain” of my own tortured statistics students

• Discuss the differences between statistics and mathematics

• Talk about how we need each other – desperately!!!

Last Quote

“To Understand God’s Thoughts We Must Study Statistics, for These Are the Measure of his Purpose”