example of models for the study of change

Example of Models for the Study of Change

David A. Kenny

December 15, 2013

2

Example DataAn Honors Thesis done by Allison Gillum of

Skidmore College supervised by John Berman on the effects of an semester-long class on the Environment on Environmental Responsible Behaviors.

Pretest-Posttest Design

41 Treated and 199 Controls

2 Treated classes and 8 Control classes. No clustering effect due to class.

Outcome: Environmentally Responsible Behaviors (ERB), a 12 item scale ranging for 1 to 7. For latent variable analyses, 3 parcels of 4 items were created.

3

Models• Models

– Controlling for Baseline• Simple• Allowing for Unreliability at Time 1

– Change Score Analysis• Raykov• LCS• Kenny-Judd

– Standardized Change Analysis• Types

– Univariate (average of 12 items)– Latent Variable (3 parcels of 4 items)

4

Latent Variable Measurement Models

• Unconstrained– 2(9) = 13.22, p = .153– RMSEA = 0.044; TLI = .992

• Equal Loadings– 2(11) = 18.63, p = .068– RMSEA = 0.054; TLI = .988• The equal loading model has

reasonable fit.

5

Pretest Difference• Mean for Controls: 4.79• Mean for Treateds: 5.21• A mean difference of 0.42• t(238) = 3.191, p = .002• d = 0.64, a moderate effect size• There is a difference at the pretest!• The mean difference on latent variable at time 1 is 0.45.

6

More on the Pretest Difference

• Likely more environmentally conscious students more likely to take an environmental course.

• Would you expect the difference to persist (CSA) or narrow (CfB)?

7

Controlling for Baseline: Univariate

•A beneficial effect of the course on the outcome: 0.3049.• Z = 3.992, p < .001• = 0.776 (expect a narrowing of the gap)•We shall see that this is the largest estimate

of the treatment effect.

8

9

CfB: Measurement Error in the Pretest

Coefficient alpha of .872 for pretestLord-Porter Correction

Convert (.872 - .2112)/( 1 - .2112) = .866

Adjusted pretest score (MT is the mean for the Treated and MC is the mean for the controls):

(X1 – MT) + MT

(X1 – MC) + MC

b = 0.2569, Z = 3.3308, p < .001

10

11

Williams & Hazer Method

Set X1 = X1T + E1

Fix the variance of E1 to (1 - )sY12

or (1 - .872)(0.614) = 0.079. b = 0.2543, Z = 3.233, p = .001

(with = .897)Same estimates of b and as Lord-

Porter (standard error a bit different)!

12

13

Controlling for Baseline: Latent Variables

–b = 0.3256, Z = 4.124, p < .001– = 0.816 (surprisingly relatively

low)–Cannot directly compare estimates

to the univariate analysis.

14

15

Change Score Analysis: Univariate–All 3 methods (see next 3 slides) show a

beneficial effect:• 0.2105, Z = 2.619, p = .009

–Smallest effect of any analysis,–Note that the Treateds improve (0.0874),

and the Controls decline (-0.1231).

16

Raykov

17

LCS

18

Kenny-Judd

19

Change Score Analysis: Latent Variables

–All 3 methods show a beneficial effect 0.2435, Z = 2.964, p < .003

–Again, you cannot directly compare the univariate and latent variable results.

–Smallest effect of any latent variable analysis.

20Raykov

21LCS

22

KJ

23

Standardized Change Score Analysis: Univariate Analysis

Residual variance decreases slightly over time (but not significantly, p = .31)

•Time 1: 0.59•Time 2: 0.52

Effect: .3078, Z = 2.775 , p = .006Recalibrated to units of Time 2: 0.2266 , Z = 2.826 , p = .005.

24SCSA

25SCSA-Y2

26

Standardized Change Score Analysis: Latent Variables

Residual variance decreases slightly over time (but not significantly, p = .30)

•Time 1: 0.56•Time 2: 0.52

b = 0.3653, Z = 3.116 , p = .002Units of Time 2: b = 0.2627, Z = 3.194 , p = .001

27

SCSA

28SCSA-Y2

29

Summary of Univariate Effects

CfB: 0.3049CfB with Reliability Correction:

0.2543CSA: 0.2105SCSA: 0.2266

30

Summary of Latent Variable Effects

CfB: 0.3256CSA: 0.2435SCSA: 0.2627

31

What Estimate Would I Report?

CSA Latent Variable: 0.2435(Z = 2.964, p < .003) No reason to think that the factors that created the Time 1 difference to change. Note too the variance does not change.Others might respectfully disagree.