statistics: unlocking the power of data lock 5 randomization tests dr. kari lock morgan psu 016...
TRANSCRIPT
Statistics: Unlocking the Power of Data Lock5
Randomization Tests
Dr. Kari Lock Morgan
PSU 016
11/5/14
Statistics: Unlocking the Power of Data Lock5
Extrasensory PerceptionIs there such a thing as extrasensory
perception (ESP) or a “sixth sense”?Do you believe in ESP?
Statistics: Unlocking the Power of Data Lock5
Extrasensory PerceptionOne way to test for ESP is with Zener cards:
Subjects draw a card at random and telepathically communicate this to someone who then guesses the symbol
Statistics: Unlocking the Power of Data Lock5
Extrasensory PerceptionLet’s do our own study!Make your own Zener cards:Randomly choose a symbolFind a partner, telepathically communicate
your symbol (no auditory or visual clues!), and have them guess your symbol.
Switch roles.Did you guess correctly?
Statistics: Unlocking the Power of Data Lock5
Extrasensory Perception
There are five cards with five different symbols
If there is no such thing as ESP, what proportion of guesses should be correct?
Because there are 5 cards, each person has a 1/5 chance of guessing correctly each time, if ESP does not exist.
H0: p = 1/5Ha: p > 1/5
Statistics: Unlocking the Power of Data Lock5
Extrasensory PerceptionStatistics vary from sample to sample: even
if the population proportion is 1/5, not every sample proportion will be exactly 1/5
How do we determine when a sample proportion is far enough above 1/5 to provide evidence of ESP?
More general: How do we determine when a sample statistic is far enough away from H0 to be statistically significant?
Statistics: Unlocking the Power of Data Lock5
Key Question
How do we know how unusual a sample statistic would be if H0 were true?
How unusual is it to see a sample statistic as extreme as that observed, if H0 is true?
SIMULATE what would happen if H0 were true!
Statistics: Unlocking the Power of Data Lock5
ESP: Simulate!• How could we simulate what would happen,
just by random chance, if the null hypotheses were true for the ESP experiment?
Randomly choose a symbol.
• Return it to the rest, shuffle, and choose again for the (random) guess.
• Did you (randomly) get the correct symbol?
Statistics: Unlocking the Power of Data Lock5
Lots of simulations!
• We need many more simulations!
www.lock5stat.com/statkey
Statistics: Unlocking the Power of Data Lock5
ESP – Random Chance
Are our results statistically significant?
What can we conclude?
Statistics: Unlocking the Power of Data Lock5
Randomization Distribution
A randomization distribution is a collection of statistics from samples
simulated assuming the null hypothesis is true
Statistics: Unlocking the Power of Data Lock5
p-value
The p-value is the chance of obtaining a sample statistic as extreme as (or more
extreme than) the observed sample statistic, if the null hypothesis is true
Statistics: Unlocking the Power of Data Lock5
1. What kinds of statistics would we get, just by random chance, if the null hypothesis were true? (randomization distribution)
2. What proportion of these statistics are as extreme as our original sample statistic? (p-value)
Calculating a p-value
Statistics: Unlocking the Power of Data Lock5
ESP p-value
p-value = 0.247
If you were all just guessing randomly, the chance of us getting a sample proportion as high as 0.294 is 0.247.
p-value
observed statistic
Proportion as extreme as observed statistic
Distribution of statistics that would be observed, just by random chance, if H0 true
Statistics: Unlocking the Power of Data Lock5
• In a randomized experiment on treating cocaine addiction, 48 people were randomly assigned to take either Desipramine (a new drug), or Lithium (an existing drug), and then followed to see who relapsed
• Is Desipramine better than Lithium at treating cocaine addiction?
Cocaine Addiction
pD, pL: proportion of cocaine addicts who relapse after taking Desipramine or Lithium, respectively
H0: pD = pL
Ha: pD < pL
Statistics: Unlocking the Power of Data Lock5
R R R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R
R R R R R R
R R R R R R
R R R R R R
R R R R
R R R R R R
R R R R R R
R R R R R R
Desipramine Lithium
1. Randomly assign units to treatment groups
Statistics: Unlocking the Power of Data Lock5
R R R R
R R R R R R
R R R R R R
N N N N N N
RRR R R R
R R R R N N
N N N N N N
RR
N N N N N N
R = RelapseN = No Relapse
R R R R
R R R R R R
R R R R R R
N N N N N N
RRR R R R
R R R R RR
R R N N N N
RR
N N N N N N
2. Conduct experiment
3. Observe relapse counts in each group
LithiumDesipramine
10 relapse, 14 no relapse 18 relapse, 6 no relapse
1. Randomly assign units to treatment groups
10 18
24
ˆ ˆ
24.333
D Lp p
Statistics: Unlocking the Power of Data Lock5
To see if a statistic provides evidence against H0, we need to
see what kind of sample statistics we would observe,
just by random chance, if H0 were true
Measuring Evidence against H0
Statistics: Unlocking the Power of Data Lock5
• “by random chance” means by the random assignment to the two treatment groups
• “if H0 were true” means if the two drugs were equally effective at preventing relapses (equivalently: whether a person relapses or not does not depend on which drug is taken)
• Simulate what would happen just by random chance, if H0 were true…
Cocaine Addiction
Statistics: Unlocking the Power of Data Lock5
R R R R
R R R R R R
R R R R R R
N N N N N N
RRR R R R
R R R R N N
N N N N N N
RR
N N N N N N
10 relapse, 14 no relapse 18 relapse, 6 no relapse
Statistics: Unlocking the Power of Data Lock5
R R R R R R
R R R R N N
N N N N N N
N N N N N N
R R R R R R
R R R R R R
R R R R R R
N N N N N N
R N R N
R R R R R R
R N R R R N
R N N N R R
N N N R
N R R N N N
N R N R R N
R N R R R R
Simulate another randomization
Desipramine Lithium
16 relapse, 8 no relapse 12 relapse, 12 no relapse
ˆ ˆ16 12
24 240.167
LDp p
Statistics: Unlocking the Power of Data Lock5
R R R R
R R R R R R
R R R R R R
N N N N N N
RRR R R R
R N R R N N
R R N R N R
RR
R N R N R R
Simulate another randomization
Desipramine Lithium
17 relapse, 7 no relapse 11 relapse, 13 no relapse
ˆ ˆ17 11
24 240.250
D Lp p
Statistics: Unlocking the Power of Data Lock5
• Shuffle your cards and deal them into two piles. What is your sample difference in proportions?
• Why did you re-deal your cards?
• Why did you leave the outcomes (relapse or no relapse) unchanged on each card?
Cocaine Addiction
You want to know what would happen
• by random chance
• if the null hypothesis is true
Statistics: Unlocking the Power of Data Lock5
Lots of simulations!
• We need many more simulations!
www.lock5stat.com/statkey
Statistics: Unlocking the Power of Data Lock5
www.lock5stat.com/statkey
p-valueProportion as extreme as observed statistic
observed statistic
If the two drugs are equal regarding cocaine relapse rates, we have a 1.3% chance of seeing a difference in proportions as extreme as that observed.
Distribution of statistics that would be observed, just by random chance, if H0 true
Statistics: Unlocking the Power of Data Lock5
Randomization Testp-values can be calculated by randomization
distributions: Create a randomization distribution by simulating
statistics you would see, just by random chance, if H0 were true
Find the p-value as the proportion of simulated statistics as extreme as the observed statistic
This idea works for any parameter!
Statistics: Unlocking the Power of Data Lock5
Your Turn! Correlation
3.0 3.5 4.0 4.5 5.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
Malevolence Rating of Uniform
z-sc
ore
for
Pen
alty
Yar
ds
r = 0.43
NFL Teams • Do NFL teams with more malevolent uniforms get more penalty yards?
Statistics: Unlocking the Power of Data Lock5
p-value and Ha
H0: = 0Ha: > 0
Upper-tail(Right Tail)
H0: = 0Ha: < 0
Lower-tail(Left Tail)
H0: = 0Ha: ≠ 0
Two-tailed
Statistics: Unlocking the Power of Data Lock5
Summaryp-values can be calculated by randomization
distributions: Create a randomization distribution by simulating
statistics you would see, just by random chance, if H0 were true
Find the p-value as the proportion of simulated statistics as extreme as the observed statistic