jacob westfall university of colorado boulder charles m. judd david a. kenny
DESCRIPTION
Treating Stimuli as a Random Factor in Social Psychology : A New and Comprehensive Solution to a Pervasive but Largely Ignored Problem . Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny University of Colorado BoulderUniversity of Connecticut. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/1.jpg)
Treating Stimuli as a Random Factor in Social Psychology:
A New and Comprehensive Solutionto a Pervasive but Largely Ignored Problem
Jacob WestfallUniversity of Colorado Boulder
Charles M. Judd David A. KennyUniversity of Colorado Boulder University of Connecticut
![Page 2: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/2.jpg)
What to do about replicability?• Mandatory reporting of all DVs, studies, etc.?• Journals or journal sections devoted to straight
replication attempts?• Pre-registration of studies?
• Many of the proposed solutions involve large-scale institutional changes, restructuring incentives, etc.
• These are good ideas worthy of discussing, but surely not quick or easy to implement
![Page 3: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/3.jpg)
One way to increase replicability:Treat stimuli as random
• Failure to account for uncertainty associated with stimulus sampling (i.e., treating stimuli as fixed rather than random) leads to biased, overconfident estimates of effects (Clark, 1973; Coleman, 1964)
• The pervasive failure to model stimulus as a random factor is probably responsible for many failures to replicate when future studies use different stimulus samples
![Page 4: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/4.jpg)
Doing the correct analysis is easy!
• Recently developed statistical methods solve the statistical problem of stimulus sampling
• These mixed models with crossed random effects are easy to apply and are already widely available in major statistical packages (R, SAS, SPSS, Stata, etc.)
![Page 5: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/5.jpg)
Outline of rest of talk1. The problem– Illustrative design and typical RM-ANOVA analyses– Estimated type 1 error rates
2. The solution– Introducing mixed models with crossed random
effects for participants and stimuli– Applications of mixed model analyses to actual
datasets
![Page 6: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/6.jpg)
Illustrative Design• Participants crossed with Stimuli
– Each Participant responds to each Stimulus • Stimuli nested under Condition
– Each Stimulus always in either Condition A or Condition B• Participants crossed with Condition
– Participants make responses under both Conditions
Sample of hypothetical dataset:
5 4 6 7 3 8 8 7 9 5 6 5
4 4 7 8 4 6 9 6 7 4 5 6
5 3 6 7 4 5 7 5 8 3 4 5
![Page 7: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/7.jpg)
Typical repeated measures analyses (RM-ANOVA)
MBlack MWhite Difference
5.5 6.67 1.17
5.5 6.17 0.67
5.0 5.33 0.33
5 4 6 7 3 8 8 7 9 5 6 5
4 4 7 8 4 6 9 6 7 4 5 6
5 3 6 7 4 5 7 5 8 3 4 5
How variable are the stimulus ratings around each of the participant means? The variance is lost due to the aggregation
“By-participant analysis”
![Page 8: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/8.jpg)
Typical repeated measures analyses (RM-ANOVA)
5 4 6 7 3 8 8 7 9 5 6 5
4 4 7 8 4 6 9 6 7 4 5 6
5 3 6 7 4 5 7 5 8 3 4 5
4.00 3.67 6.33 7.33 3.67 6.33 8.00 6.00 8.00 4.00 5.00 5.33
Sample 1 v.s. Sample 2
“By-stimulus analysis”
![Page 9: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/9.jpg)
Simulation of type 1 error rates for typical RM-ANOVA analyses
• Design is the same as previously discussed• Draw random samples of participants and stimuli– Variance components = 4, Error variance = 16
• Number of participants {10, 30, 50, 70, 90}∈• Number of stimuli {10, 30, 50, 70, 90}∈• Conducted both by-participant and by-stimulus
analysis on each simulated dataset• True Condition effect = 0
![Page 10: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/10.jpg)
![Page 11: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/11.jpg)
Type 1 error rate simulation results• The exact simulated error rates depend on the
variance components, which although realistic, were ultimately arbitrary
• The main points to take away here are:1. The standard analyses will virtually always show
some degree of positive bias2. In some (entirely realistic) cases, this bias can be
extreme3. The degree of bias depends in a predictable way on
the design of the experiment (e.g., the sample sizes)
![Page 12: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/12.jpg)
The old solution: Quasi-F statistics• Although quasi-Fs successfully address the
statistical problem, they suffer from a variety of limitations– Require complete orthogonal design (balanced factors)– No missing data– No continuous covariates– A different quasi-F must be derived (often laboriously)
for each new experimental design – Not widely implemented in major statistical packages
![Page 13: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/13.jpg)
The new solution: Mixed models• Known variously as:– Mixed-effects models, multilevel models, random
effect models, hierarchical linear models, etc.• Most social psychologists familiar with mixed
models for hierarchical random factors– E.g., students nested in classrooms
• Less well known is that mixed models can also easily accommodate designs with crossed random factors– E.g., participants crossed with stimuli
![Page 14: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/14.jpg)
![Page 15: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/15.jpg)
Grand mean = 100
![Page 16: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/16.jpg)
![Page 17: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/17.jpg)
MeanA = -5 MeanB = 5
![Page 18: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/18.jpg)
![Page 19: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/19.jpg)
ParticipantIntercepts5.86
7.09
-1.09
-4.53
![Page 20: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/20.jpg)
![Page 21: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/21.jpg)
Stim. Intercepts: -2.84 -9.19 -1.16 18.17
![Page 22: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/22.jpg)
![Page 23: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/23.jpg)
ParticipantSlopes3.02
-9.09
3.15
-1.38
![Page 24: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/24.jpg)
![Page 25: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/25.jpg)
Everything else = residual error
![Page 26: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/26.jpg)
![Page 27: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/27.jpg)
The linear mixed-effects modelwith crossed random effects
Fixed effects Random effects
![Page 28: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/28.jpg)
The linear mixed-effects modelwith crossed random effects
Intercept Slope
6 parameters
![Page 29: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/29.jpg)
Fitting mixed models is easy: Sample syntaxlibrary(lme4)model <- lmer(y ~ c + (1 | j) + (c | i))
proc mixed covtest;class i j;model y=c/solution;random intercept c/sub=i type=un;random intercept/sub=j;run;
MIXED y WITH c /FIXED=c /PRINT=SOLUTION TESTCOV /RANDOM=INTERCEPT c | SUBJECT(i) COVTYPE(UN) /RANDOM=INTERCEPT | SUBJECT(j).
R
SAS
SPSS
![Page 30: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/30.jpg)
Mixed models successfully maintain the nominal type 1 error rate (α = .05)
![Page 31: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/31.jpg)
Applications to existing datasets1. Representative simulated dataset (for
comparison)2. Afrocentric features data (Blair et al., 2002,
2004, 2005)3. Shooter data (Correll et al., 2002, 2007)4. Psi / Retroactive priming data (Bem)– Forward-priming condition (classic evaluative
priming effect)– Reverse-priming condition (psi condition)
![Page 32: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/32.jpg)
Comparison of effectsbetween RM-ANOVA and mixed model analyses
Dataset RM-ANOVA (by-participant) Mixed model Stimulus ICC
F ratio D.F. p F ratio D.F. p
Simulated example
30.48 (1, 29) <.001 9.11 (1, 38.52) .005 r = 0.191
Shooter data 57.89 (1, 35) <.001 3.39 (1, 48.1) .072 r = 0.317
Afrocentric features data
6.40 (1, 46) .015 4.33 (1, 51.1) .043 r = 0.113
Bem (2011)Forward-priming condition
22.18 (1, 98) <.001 14.59 (1, 46.91) .029 Targets: r = 0.349Primes: r = 0.035
Bem (2011) Reverse-priming condition
6.60 (1, 98) .012 2.34 (1, 27.58) .136 Targets: r = 0.292Primes: r = 0.0
![Page 33: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/33.jpg)
Comparison of effectsbetween RM-ANOVA and mixed model analyses
Dataset RM-ANOVA (by-participant) Mixed model Stimulus ICC
F ratio D.F. p F ratio D.F. p
Simulated example
30.48 (1, 29) <.001 9.11 (1, 38.52) .005 r = 0.191
Shooter data 57.89 (1, 35) <.001 3.39 (1, 48.1) .072 r = 0.317
Afrocentric features data
6.40 (1, 46) .015 4.33 (1, 51.1) .043 r = 0.113
Bem (2011)Forward-priming condition
22.18 (1, 98) <.001 14.59 (1, 46.91) .029 Targets: r = 0.349Primes: r = 0.035
Bem (2011) Reverse-priming condition
6.60 (1, 98) .012 2.34 (1, 27.58) .136 Targets: r = 0.292Primes: r = 0.0
![Page 34: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/34.jpg)
Comparison of effectsbetween RM-ANOVA and mixed model analyses
Dataset RM-ANOVA (by-participant) Mixed model Stimulus ICC
F ratio D.F. p F ratio D.F. p
Simulated example
30.48 (1, 29) <.001 9.11 (1, 38.52) .005 r = 0.191
Shooter data 57.89 (1, 35) <.001 3.39 (1, 48.1) .072 r = 0.317
Afrocentric features data
6.40 (1, 46) .015 4.33 (1, 51.1) .043 r = 0.113
Bem (2011)Forward-priming condition
22.18 (1, 98) <.001 14.59 (1, 46.91) .029 Targets: r = 0.349Primes: r = 0.035
Bem (2011) Reverse-priming condition
6.60 (1, 98) .012 2.34 (1, 27.58) .136 Targets: r = 0.292Primes: r = 0.0
![Page 35: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/35.jpg)
Conclusion• Many failures of replication are probably due to
sampling stimuli and the failure to take that into account
• Mixed models with crossed random effects allow for generalization to future studies with different samples of both stimuli and participants
![Page 36: Jacob Westfall University of Colorado Boulder Charles M. Judd David A. Kenny](https://reader036.vdocuments.us/reader036/viewer/2022062315/568161ab550346895dd16878/html5/thumbnails/36.jpg)
The end
Further reading:Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely
ignored problem. Journal of personality and social psychology, 103(1), 54-69.