psy 1950 confidence and power december, 1 2008

PSY 1950Confidence and PowerDecember, 1 2008

Requisite Quote“The picturing of data allows us to be sensitive not

only to the multiple

hypotheses that we hold, but to the many more we have not yet thought of, regard as

unlikely, or think impossible.”– Tukey, 1974

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Point Estimation vs. Interval Estimation

• Confidence intervals estimate parameters with intervals instead of simply points

0

20

40

60

80

100

120

140

A B

Dependent variable

0

20

40

60

80

100

120

140

A B

Dependent variable

Reliability• Confidence intervals indicate the reliability of an estimate

0

20

40

60

80

100

120

140

A B

Dependent variable

0

20

40

60

80

100

120

140

A B

Dependent Variable

Confidence Level• Confidence intervals are a function of the desired precision

0

20

40

60

80

100

120

140

A B

Dependent variable

0

20

40

60

80

100

120

140

A B

Dependent Variable

Confidence Interval• Definition

– “For a given proportion p (where p is the confidence level), a confidence interval for a population parameter is an interval that is calculated from a random sample of an underlying population such that, if the sampling was repeated numerous times and the confidence interval recalculated from each sample according to the same method, a proportion p of the confidence intervals would contain the population parameter in question”

http://www.ruf.rice.edu/~lane/stat_sim/conf_interval/

Correct Interpretation• e.g., “If we replicated our experiment, the calculated confidence intervals would contain the true population mean 95% of the time.”

• e.g., “The confidence interval represents values for the population parameter for which the difference between the parameter and the observed estimate is not statistically significant at the 5% level."

Incorrect Interpretation• Confidence intervals do NOT reflect the probability that the parameter falls within an estimated range– Wrong: “There is a 95% chance that the actual mean group difference is between 2-3.”

• The true mean is fixed, not variable: it either falls inside or outside a particular range– e.g., what is the probability that 2 is between 3-4?

“This is not just quibble”• Imagine CIs constructed from two samples from the same population– 1st sample’s 70% CI: 1-2•70% chance that the mean is between 1-2

– 2nd sample’s 70% CI: 2-3•30% chance that the mean is less than 2 or greater than 3

– These are incompatible!

0

1

2

3

4

1 2

Sample

Dependent Variable

Eyeballing Interval Estimates

• Question matters– Do you care about estimated values or differences of estimated values?

• Values matters– Note actual values of point estimates (and relevant differences)

– Note actual values of interval estimates (and relevant differences)

– SE, SD, CI?

• Context matters– How you interpret CI on individual group/condition means depends on experimental design

Independent Measures

0

1

2

3

4

A B Dif

Group

Dependent Variable

• Difference CI can be inferred from mean CIs– Always larger– If group CIs are similar, larger by factor of √2

Dependent Measures

• Difference CI cannot be inferred from mean CIs– Depends on correlation– Usually smaller than mean CIs (r is usually positive and large)

0

1

2

3

4

Pre Post Dif

Condition

Dependent Variable

Rules of Thumb: 95% CI• So long as ns are at least 10 and one error is not greater than twice the other:– If proportion overlap is .5 or less, t-test is significant at = .05

– If there is no overlap, t-test is significant at = .01

0

1

2

3

A B

Group

Dependent Variable 0

1

2

3

A B

Group

Dependent Variable

Rules of Thumb: SE• So long as ns are at least 10 and one error is not greater than twice the other:– If proportion gap is 1 or greater, t-test is significant at = .05

– If proportion gap is 2 or greater, t-test is significant at = .01

0

1

2

3

A B

Group

Dependent Variable 0

1

2

3

A B

Group

Dependent Variable

CI Calculation

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Example

Dependent Measures

Additional Resources

CIs and Replication• Given a sample mean and 95% CI, what is the probability that a repetition of that experiment, with an independent sample of participants, would give a mean that falls within the original CI?– Not .95– Actually .834

• Why?– Probability that CI “captures” population mean depends on:• Deviation of sampled mean from population mean

– Probability that CI “captures” replicant mean depends on:• Deviation of original (sample) mean from population mean

• Deviation of replicant (sample) mean from population mean

0

10

20

30

40

50

60

70

80

90

100

Independent experiments

Dependent variable

Power• The probability of correctly rejecting a false null hypothesis

• The probability of not making a Type II error

• Power depends on– Effect size– Sample size– Alpha

Significance = Effect size Sample size

Prospective Power Analysis• Question: How many subjects do I need to have reasonable odds of getting a significant effect?

• Calculation– Determine desired power

•Usually .8

– Estimate effect size•Prior research•Theoretical considerations

– Solve for nhttp://wise.cgu.edu/power/power_applet.html

Retrospective Power Analysis

• Question: If the true effect size was the one I found, what was the power of my experiment?

• Calculation– Use obtained effect size and sample size

• Controversial/wrong claim: If you had high retrospective power and nonsignificant results, the null hypothesis is probably true– It is not possible to have high retrospective power and reject the null•With p = .05, power = .5•With p > .05, power < .5