Hypothesis Testing for Simulation 1

hypothesis testing with special focus on

simulation

Hypothesis Testing for Simulation 2

Hypothesis Test answers yes/no question with some statistical certainty

H0 = default hypothesisis a statement

Ha = alternate hypothesisis the precise opposite

Hypothesis Testing for Simulation 3

X = test statistic (RANDOM!)sufficient (uses all avail. data)often Z, T, N are used as notation

FX = its probability distribution

a = P[reject H0 | H0 true]

Hypothesis Testing for Simulation 4

ca = critical region for a

a = P[X in ca | H0]

a is our (controllable) risk

Hypothesis Testing for Simulation 5

TWISTED LOGIC

We WANT to reject H0 and conclude Ha, so...We make a very small, so...If we can reject, we have strong

evidence that Ha is true

This construct often leads to inconclusive results“There is no significant statistical

evidence that...”

Hypothesis Testing for Simulation 6

IMPORTANT

Inability to reject <> H0 true

Hypothesis Testing for Simulation 7

POWER OF THE TEST

b = P[X not in ca | Ha]

1-b = P[correctly rejecting]

Hypothesis Testing for Simulation 8

VENACULAR

a is type I errorProbability of incorrectly rejecting

b is type II errorProbability of incorrectly missing the opportunity to reject

Hypothesis Testing for Simulation 9

UNOFFICIAL VENACULAR

type III error – answered the wrong question

type IV error – perfect answer delivered too late

Hypothesis Testing for Simulation 10

EXAMPLE!

Dial-up ISP has long experience & knows...

50

)]([

STHdownloadE

Hypothesis Testing for Simulation 11

We get DSL, observe 12 samples

9.11ˆ

0.42

X

Hypothesis Testing for Simulation 12

IS DSL FASTER?

H0: mDSL = 50

Ha: mDSL < 50

test with P[type I] = 0.01

Hypothesis Testing for Simulation 13

PROBABILITY THEORY

Z ~ tn-1Must know the probability distribution of the

test statistic IOT construct critical region

n

XZ

ˆ

Hypothesis Testing for Simulation 14

for n = 12, a = 0.01, ca = -2.718

99% of the probabilityabove -2.718

Hypothesis Testing for Simulation 15

our test statistic-2.33

33.2

129.11

)5042(

Hypothesis Testing for Simulation 16

0.021 called the p-value

Given H0, we expect to see a test statistic as extreme as Z roughly 2% of the time.

-2.718(0.01)

-1.796(0.05)-2.33

(0.021)

Hypothesis Testing for Simulation 17

CONFIDENCE INTERVALS

la ua

For a given aP[la <= m <= ua] = 1-a

mBased on the sampleSo they are RANDOM!

Hypothesis Testing for Simulation 18

GOODNESS-OF-FIT TEST

Discrete, categorized dataRolls of diceMiss distances in 5-ft. increments

H0 assumes a fully-specified probability modelHa: the glove does not fit!

Hypothesis Testing for Simulation 19

TEST STATISTIC

2

1

22 ~

exp

)exp(

n

i i

ii

ected

ectedobsX

“chi-squared distribution with gnu degrees of freedom”

Hypothesis Testing for Simulation 20

n = observations - estimated param

Did you know... if Zi~N(0, 1), then

Z12+ Z2

2+...+ Zn2 ~ cn

2

Hypothesis Testing for Simulation 21

CELLS

H0 always results in a set of category cells with expected frequencies

EXAMPLECoin is tossed 100 timesH0: Coin Fair

Hypothesis Testing for Simulation 22

CELLS AND EXPECTED FREQUENCIES

  EXPECT

H 50

T 50

Hypothesis Testing for Simulation 23

EXAMPLE

Cannon places rounds around a targetH0: miss distance ~ expon(0.1m)

Record data in 5m intervals(0-5), (5-10), ...(25+)

Hypothesis Testing for Simulation 24

EXPONENTIALS

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 10 20 30 40 50 60

F(x)

f(x)

x

x

exf

exF

)(

1)(

E(X)=1/l

Hypothesis Testing for Simulation 25

RESULTS

RIGHT OBS 1-exp(-0.1x) PROB EXPECT (OBS-EXPECT)^2

    0.00      

5.00 30 0.39 0.39 39.35 2.22

10.00 17 0.63 0.24 23.87 1.97

15.00 21 0.78 0.14 14.47 2.94

20.00 11 0.86 0.09 8.78 0.56

25.00 11 0.92 0.05 5.33 6.05

30+ 10 1.00 0.08 8.21 0.39

           

  100.00     14.14

Hypothesis Testing for Simulation 26

0

5

10

15

20

25

30

35

40

45

0 10 20 30 40

observed

expected

Hypothesis Testing for Simulation 27

TEST RESULTS

Degrees of Freedom6 cells0 parameters estimatedn = 6

For the c62 distribution, the p-

value for 14.14 is about p=0.025

REJECT at any a > 0.025

Hypothesis Testing for Simulation 28

DIFFERENT H0

H0: the miss distances are exponentially distributed

Ha: the exponential shape is incorrect

We estimate the parameter, we lose one degree of freedom

Hypothesis Testing for Simulation 29

RESULTS 2

LEFT RIGHT OBS 1-exp(-0.0738x) PROBEXPE

CT (OBS-EXPECT)^2

      0.00      

0.00 5.00 30 0.31 0.31 30.86 0.02

5.00 10.00 17 0.52 0.21 21.34 0.88

10.00 15.00 21 0.67 0.15 14.75 2.65

15.00 20.00 11 0.77 0.10 10.20 0.06

20.00 25.00 11 0.84 0.07 7.05 2.21

25.00 30+ 10 1.00 0.16 15.80 2.13

             

        7.95

Hypothesis Testing for Simulation 30

0

5

10

15

20

25

30

35

0 10 20 30 40

observed

expected

Hypothesis Testing for Simulation 31

n = 5

p-value for 7.83 is larger than 0.05

CANNOT REJECT

CONCLUSION?

Hypothesis Testing for Simulation 32

SIMULATION vs. STATISTICS

StatisticsSample is fixed and givenConclusion is unknownSignificance is powerful

SimulationSample is arbitrarily largeConclusion is knownWe need another thought about what is

meaningful

Hypothesis Testing for Simulation 33

SAMPLE SIZE EFFECT

m = 100s = 10

Hypothesis Testing for Simulation 34

HOW LARGE IS A DIFFERENCE BEFORE IT IS MEANINGFUL?

mu lower upper sigma

10 101.0468 98.16152 103.9321 5.547101100 101.3426 99.8384 102.8468 9.144828500 101.0861 100.3455 101.8266 10.06773

1000 100.8007 100.2762 101.3253 10.0847

97

98

99

100

101

102

103

104

105

0 200 400 600 800 1000

x-bar

lower

upper

Hypothesis Testing for Simulation 35

SUMMARY

You probably knew the mechanics of HT

You might have a new perspective