reintroduction to factorial designs (re-)introduction to factorial designs 5 patterns of factorial...

Reintroduction to Factorial Designs

• (re-)Introduction to factorial designs

• 5 patterns of factorial results

• Understanding main effects

• Research Hypotheses for factorial designs

• Other ways of thinking about interactions

• Practice

Type of Therapy

Initial Diagnosis Group Individual

General clients diagnosed w/ clients diagnosed w/ Anxiety general anxiety who general anxiety who

received group therapy received individual therapy

Social clients diagnosed w/ clients diagnosed w/ Anxiety social anxiety who social anxiety who

received group therapy received individual therapy

Participants in each “cell” of this design have a unique combination of IV conditions.

Effects examined by a factorial design

There are always THREE effects (IVs) examined ..

1 -- the interaction of the two IVs -- how they jointly relate to DV

2 -- the main effect of the one IV -- how it relates to the DV independently of the interaction and the other main effect

3 -- the main effect of the other IV -- how it relates to the DV independently of the interaction and the other main effect

For the example…

1 -- the “interaction” of Initial Diagnosis & Type of Therapy

2 -- the “main effect” of Initial Diagnosis

3 -- the “main effect” of Type of Therapy

The difficult part of learning about factorial designs is the large set of new terms that must be acquired. Here’s a summary;

cell means -- the mean DV score of all the folks with a particular combination of IV treatments

marginal means -- the mean DV score of all the folks in a particular condition of the specified IV (aggregated across conditions of the other IV)

Main effects involve the comparison of marginal means.

Simple effects involve the comparison of cell means.

Interactions involve the comparison of simple effects.

Identifying Cell Means and Marginal Means

Type of Therapy

Initial Diagnosis Group Individual

General Anxiety 50 50 50

Social Anxiety 90 10 50

70 30

Cell means mean DV of subjects in each design cell

Marginal means average mean DV of all subjects in one condition of an IV

Identifying Main Effects -- difference between the marginal meansof that IV (ignoring the other IV)

Type of TherapyInitial Diagnosis Group Individual



70 30

Main effect of Initial Diagnosis

Main effect of Type of Therapy

Identifying Simple Effects -- cell means differences between conditions of one IV for a specific level of the other IV




a b

Simple effects of Type of Therapy for each Initial Diagnosis

1 Simple effect of Type of Therapy for general anxiety patients

2 Simple effect of Type of Therapy for social anxiety patients

Identifying Simple Effects -- cell means differences between conditions of one IV for a specific level of the other IV




a b

Simple effects of Initial Diagnosis for each Type of Therapy

a Simple effect of Initial Diagnosis for group therapy

b Simple effect of Initial Diagnosis for individual therapy

5 basic factorial data patterns - 3 have interactions – 2 do not

#1 Task Presentation Paper ComputerTask Difficulty

Easy 90 > 70 simple effects are

Hard 40 < 60 opposite directions

There is an interaction of Task Presentation and Task Difficulty as they relate to performance. Easy tasks are performed better using paper than using computer (90 vs. 70), whereas hard tasks are performed better using the computer than using paper (60 vs. 40).

#1a – an important variation of this data pattern

I call this, “The effect that isn’t anywhere!”• There is a significant interaction(p < .05). • But – neither simple effect is statistically significant !?!?!

Task Presentation Paper ComputerTask Difficulty

Easy 90 = 70 .

Hard 40 = 60

simple effects look like they’re in

opposite directions, but are both nulls!!!

Remember! •An interaction is that the SEs are “different from each other”. •The significant interaction tells us the 90>70 simple effect is different from the 40<60 simple effect.

#2

Task Presentation Paper Computer

Task Difficulty

Easy 90 = 90 one simple effect “null”

Hard 40 < 70 one simple effect

There is an interaction of Task Presentation and Task Difficulty as they relate to performance. Easy tasks are performed equally well using paper and using the computer (90 vs. 90), however, hard tasks are performed better using the computer than using paper (70 vs. 40).

#3 Task Presentation

Paper ComputerTask Difficulty

Easy 80 < 90 simple effects in the same

direction,

Hard 40 < 70 but of different sizes

There is an interaction of Task Presentation and Task Difficulty as they relate to performance. Performance was better using the computer than using paper, however this effect was larger for hard tasks (70 vs. 40) than for easy tasks (90 vs. 80).

There are the two basic patterns of NON-interactions



Easy 30 < 50 both simple effects are in thesame direction and are

Hard 50 < 70 the same size

There is no interaction of Task Presentation and Task Difficulty as they relate to performance. Performance is better for computer than for paper presentations (for both Easy and Hard tasks).

Notice the main effects will be descriptive.



Easy 50 = 50 both simple effects

Hard 70 = 70 are nulls

There is no interaction of Task Presentation and Task Difficulty as they relate to performance. Performance is the same for computer and paper presentations (for both Easy and Hard tasks).

Notice the main effects will be descriptive.

Understanding Main Effects

Patterns of data that include main effects can be identified by looking at the differences among the marginal means for a specific IV (the main effect of each IV must be examined and described separately !!!)

• When there is an interaction, each main effect (null or significant) must be carefully examined to determine if that main effect is

•“descriptive” (unconditional, that is, descriptive for all levels of the other IV) or is

• “potentially misleading (conditional, that is, descriptive for only some or none of the levels of the other IV)

• You must determine whether the pattern of each main effect (direction of any difference between the marginal means) is equivalent to each of the corresponding simple effects of that variable at the various levels of the other I

Are Main Effects Important ??

It is not uncommon to hear the advice to “ignore main effects if there is an interaction.”

My best guess is that this is based on the correct idea that the pattern of some interactions can render the pattern of one or both main effects to be potentially or completely misleading.

However, it is also possible that there can be an interaction and that one or both of the main effects can be descriptive.

Discerning whether main effects are descriptive or misleading is a critical step in the examination of data from a factorial design! You must ensure that the reader has a thorough understanding of the pattern of your data!

You must give a complete accounting of each of the three effects involved in the factorial design, the interaction and each of the main effects!

Interpreting main effects … When there is an interaction, the pattern of the interaction may influence the interpretability (generality) of the description of the marginal means.


Task Difficulty There is a main effect for Easy 80 < 90 Task Presentation, overall

performance was better using computer presenta-

Hard 40 < 70 tion than using paper presentation.

60 < 80

Notice: that the pattern of the main effect is consistent with both the simple effect of Task Presentation for easy tasks and the simple effect of Task Presentation for hard tasks.

Another example …


Task Difficulty

Easy 90 = 90

Hard 40 < 70

65 < 80

There is a main effect for Task Presentation, overall performance was better using computer presentation than using paper presentation. However, this pattern is descriptive for hard tasks, but not for easy tasks, for which there was no simple effect of Task Presentation.

There is a main effect for Task Presentation, overall performance was better using computer presentation than using paper presentation.

Yet another example …


Task Difficulty

Easy 80 > 60

Hard 20 < 70

50 < 65

There is a main effect for Task Presentation, overall performance was better using computer presentation than using paper presentation. However, this pattern is descriptive for hard tasks, but not for easy tasks, for which performance was better using paper presentations than using computer presentation.

There is a main effect for Task Presentation, overall performance was better using computer presentation than using paper presentation.

“Null” main effects can also be misleading….

Task Presentation Paper ComputerTask Difficulty

Easy 90 > 70

Hard 40 < 60

65 = 65

There is no main effect for Task Presentation, overall performance was equivalent using computer presentation and using paper presentation. However, this pattern is descriptive for neither hard tasks, for which computer presentations worked better than paper, nor for easy tasks, for which performance was better using paper presentations than using computer presentation.

There is no main effect for Task Presentation, overall performance was the same using computer and paper presentation.

1 < vs. > simple effects in opposite directions

2. = vs. < one null simple effect and one simple effect

3. < vs. < simple effects in same direction, but different

sizes

4. < vs. < simple effects of the same size in the same direction

5. = vs. = both null simple effects

We can rearrange the 5 basic patterns of results from a 2x2 Factorial to help us think about interactions and descriptive/misleading main effects

Interaction

-- simple effects of different size and/or direction

Misleading main effects

Descriptive main effects

No Interaction

-- simple effects are null or same size

Related to this is the very important issue of whether or not the main effects that are part of the 2-way design “mean anything to us” ???

It all goes back to “representation & inference” !!!

Remember – the purpose of any design condition or value is to represent some population so we can infer that the difference between those conditions or values in the design represent differences between the populations we really care about!

The “cells” in the 2-way each represent a specific population and so, comparisons between them are comparisons between out target populations.

But the means/values of all the main effects are “aggregates” – who do they represent???

Here’s an example to help to consider this…

We know what population is represented by each of the four cell means!

What about the marginal mean for “Paper Presentation” the aggregate of Easy & Hard Difficulty…

65

Does it represent “any difficulty”… “medium difficulty” ???

What about the marginal mean “Hard Task Difficulty”???

What population is represented by the aggregate of Paper & Computer Task Presentations????

55

So, to summarize…

When are main effects “interesting”?

1.When the marginal means represent a target population• When the “mix” or “aggregate” of the cell means does

represent some well-defined population of interest• Then the marginal means represent “a finding for that

population”• Caution other populations might have different patterns

2.When the pattern of the marginal means is descriptive of the pattern in the corresponding simple effects

• When the basic pattern is the same for each of the different groups or conditions of the other IV

• Then the marginal means represent a “general finding”• Caution carefully separate “general finding” from

“finding about a specific population”

RH: for Factorial Designs

Research hypotheses for factorial designs may include

• RH: for main effects

• involve the effects of one IV, while ignoring the other IV

• tested by comparing the appropriate marginal means

• RH: for interactions

• usually expressed as “different differences” -- differences between a set of simple effects

• tested by comparing the results of the appropriate set of simple effects

• That’s the hard part -- determining which set of simple effects gives the most direct test of the interaction RH:

Sometimes the Interaction RH: is explicitly stated

• when that happens, one set of SEs will provide a direct test of the RH: (the other won’t)

This is most directly tested by inspecting the simple effect of paper vs. computer presentation for easy tasks, and comparing it to the simple effect of paper vs. computer for hard tasks.

Here’s an example:

Easy tasks will be performed equally well using paper or computer presentation, however, hard tasks will be performed better using computer presentation than paper.

Presentation Comp PaperTask Diff.

Easy

Hard

=

>

Sometimes the set of SEs to examine use is “inferred” ...Often one of the IVs in the study was used in previous research, and the other is “new”.

• In this case, we will usually examine the simple effect of the “old” variable, at each level of the “new” variable

•this approach gives us a clear picture of the replication and generalization of the “old” IV’s effect.

e.g., Previously I demonstrated that computer presentations lead to better learning of statistical designs than does using a conventional lecture. I would like to know if the same is true for teaching writing.

Let’s take this “apart” to determine which set of SEs to use to examine the pattern of the interaction...

Previously I demonstrated that computer presentations lead to better learning of statistical designs than does using a conventional lecture. I would like to know if the same is true for teaching writing.

Here’s the design and result of the earlier study about learning stats.

Type of Instruction Comp Lecture

>

Here’s the design of the study being planned.

Type of Instruction Comp LectureTopic

Stats

Writing

What cells are a replication of the earlier study ?

So, which set of SEs will allow us to check if we got the replication, and then go on to see of we get the same results with the new topic ?

Yep, SE of Type of Instruction, for each Topic ...

Sometimes the RH: about the interaction and one of the main effects are “combined”

• this is particularly likely when the expected interaction pattern is of the > vs. > type

Here’s an example…

Group therapy tends to work better than individual therapy, although this effect is larger for patients with social anxiety than with agoraphobia.

Type of Therapy Group Indiv.Anxiety

Social

Agora.>

>

Main effect RH:

>Int. RH:

So, we would examine the interaction by looking at the SEs of Type of Therapy for each type of Anxiety.

Other ways of thinking about Interactions

Generally folks define, identify, and describe interactions in terms of “different differences” or “different simple effects,” as we have done so far…


Task DifficultyEasy 90 = 90 one simple effect “null”

Hard 40 < 70 one simple effect

However, that’s not the only way to define, identify or describe interactions…

Sometimes our hypotheses aren’t about patterns of simple effects, but … are about other kinds of mean difference patterns…

The IVs are “Training Modality” and “Testing Modality” leading to this 2x2 factorial design…

VV

Training ModalityVisual Touch

VT TT

TV

Te

stin

g M

oda

lity

To

uch

V

isua

l

Among these conditions, 2 are “intramodal” (VV & TT) & 2 are “cross-modal” (VT & TV).

RH:s for the study were…RH1: VV > TT hypothesized dif among intramodal conditionsRH2: VT > TV hypothesized dif among cross-modal conditions

Neither of which corresponds to a “simple effect” !!REM! LSDmmd can be used to compare among any of the cells!

At the time, using LSDmmd hadn’t “caught on” yet, and since neither RH: corresponds to a SE, folks would use separate t-tests to compare each cell pair – leading several to make Type II errors because of the lowered power…

VV


VT

TT

TV

Te

stin

g M

oda

lity

Cro

ss

In

tra

In this case there is an “organizational” solution…Just re-label the IVs…“Training Modality” Vision vs. Touch & “Testing Modality” Intramodal vs. Cross-modal then…

The “significant interaction that isn’t anywhere” we’ve discussed!

RH1: VV > TT SE of Training Modality for Intramodal testsRH2: VT > TV SE of Training Modality for Cross-modal tests

Per

form

ance

99.6%


26.2 %

Te

stin

g M

oda

lity

To

uch

V

isua

l25.6 %

24.8 %

… which can be directly & completely tested using the 6 pairwise comparisons among the 4 conditions.

VV VT TV TT

Another Example – same research area…

This was the common design for studying intra- and cross-modal memory with the usual RH: VV > VT > TV = TT

After several studies, someone noticed that these conditions define a factorial…

99.6%


26.2 %

Te

stin

g M

oda

lity

To

uch

V

isua

l

25.6 %

24.8 %

There was an interaction!

There was a (misleading) main effect of Training Modality.

There was a (misleading) main effect of Testing Modality.

However interesting and informative was the idea from the significant interaction, that “performance is the joint effect of Training and Testing Modalities” – none of these “effect tests” give a direct test of the RH:

The set of pairwise comparisons gives the most direct RH test!!!

Notice how the very large VV cell mean “drives” both main effects (while ensuring they will each be misleading) as well as driving the interaction!?!

Here’s yet another way of thinking about interactions… we’ll need an example!!

Brownies – great things… worthy of serious theory & research!!!

The usual brownie is made with 4 blocks of chocolate and 2 cups of sugar. Replicated research tells us that the average rating of brownies made with this recipe is about 3 on a 10-point scale.

My theory? People don’t really like brownies! What they really like is fudge! So, goes my theory, making brownies more “fudge-like” will make them better liked.

How to make them more fudge-like, you ask?

Add more sugar & more chocolate!!!

So, we made up several batches of brownies and asked people to taste a standardized amount of brownie after rinsing their mouth with water, eating an unsalted saltine cracker and rinsing their mouth a second time. We used the same 10-point rating scale; 1 = this is the worst plain brownie I’ve ever had, 10=this is the best plain brownie I’ve ever had.

Our first study:

2-cups of sugar 4-cups of sugar

3 5

So, far so good!

Our second study:

4 blocks of choc.

3 2

8 blocks of choc.

What???? Oh – yeah! Unsweetened chocolate…

Then the argument started..

One side: We have partial support for the theory – addingsugar helps, but adding chocolate hurts!!!

Other side: We have not tested the theory!!!

What was our theory?

Add more sugar & more chocolate!!! We need a better design!

4 blocks of choc.

3 2

8 blocks of choc.

2-cups of sugar

4-cups of sugar

5

What do we expect for the 4-cup & 8-block brownies?

standard brownie

+ sugar effect

+ chocolate effect

expected additive effect of choc & sugar

expected score for 4&8 brownies

3

+ 2

- 1

1

4

4 blocks of choc.

3 2

8 blocks of choc.

2-cups of sugar

4-cups of sugar 5

How do we account for this ?

9

There is a non-additive joint effect of chocolate and sugar!!!!

The joint effect of adding chocolate and sugar is not predictable as the sum of the effects of adding each!!!

Said differently, there is an interaction of chocolate and sugar that emerges when they are added simultaneously.

The effect of adding both simultaneously is 6 … not 1???

This leads to the distinction between two “kinds” of interactions…

“Augmenting” Interaction

10

# practices10 30

~FB

FB 20 45

15

The combined effect is greater than would be

expected as the additive effect!

“Interfering” Interaction

10~Aud

Aud

~Rew Rew

25 15

20

The combined effect is less than would be expected as

the additive effect!

Practice effect = 5Feedback effect = 10Expected additive effect = 15Joint effect = 35

“Augmenting” Interaction

45

Reward effect = 10Audience effect = 15Expected additive effect = 25Joint effect = 5

“Interfering” Interaction“Interfering” Interaction“Interfering” Interaction

“Describing a pattern of data that includes an interaction” vs.“Describing the Interaction in a pattern of data”

90 70 50 30

Paper Computer

Task Presentation

The pattern of data shown the figure demonstrate that while Task Presentation has no effect for Easy tasks, for Hard tasks, those using Computer did better than when using Paper.

This is “a description of a pattern of data that includes an interaction”

Technically, it would be wrong to say that “The interaction shown in the figure demonstrates that while Task Presentation has no effect for Easy tasks, for Hard tasks, those using Computer did better than when using Paper.

In order to “describe the interaction effect” we have to isolate the “interaction effect” from the main effects…

Easy

Hard

The process, called “mean polishing,” involves residulaizing the data for the main effects, leaving the interaction effect… Presentation Paper Comp means row effect

Easy 90 90 90 +15 Hard 50 70 60 -15 means 70 80 75 grand mean

col effect -5 +5

Correcting for row effects (subtract +/- 15)

Presentation Paper Comp

Easy 75 75 Hard 65 85

Correcting for column effects (subtract +/- 5)


Easy 80 70 Hard 70 80

Correcting for Grand Mean (subtract 75)


Easy 5 -5 Hard -5 5

10 5 0 -5 -10

Paper Computer

Task Presentation

Easy

Hard

The proper description of “the interaction effect” is

The interaction shown in the figure demonstrates that for Easy tasks those using Paper performed better than those using Computer, however, for Hard tasks, those using Computer performed better than those using Paper.

Hard

Easy

Hard

Looked at in this way, interactions differ in only 2 ways…

Which group has “increase” and which had “decrease”

Easy

Hard

vs.Easy

Hard

The “strength” of the interaction effect…

Easy

Hard

Easy

Hard

Easy

Hard

Easy

Hard

Easy

Hard

Easy

Hard

Easy

Hard

EasyEasy

Hard

Easy

Hard

null small medium large

Practice #1


General Anxiety 60 40

Social Anxiety 70 80

• Interaction described as Initial Diagnosis effect for each type of Therapy

• Interaction describes as Type of Therapy effect for each Initial Diagnosis

• Main effect of Type of Therapy

• Main effect of Initial Diagnosis

Practice #2




• Interaction described as Initial Diagnosis effect for each type of Therapy

• Interaction describes as Type of Therapy effect for each Initial Diagnosis

• Main effect of Type of Therapy

• Main effect of Initial Diagnosis

Practice #3




RH#1: Those receiving Individual Therapy have higher scores and this effect is stronger for those with Social Anxiety.

RH#2: Previous research showed that those with General Anxiety responded better to Group Therapy than did those with Social Anxiety. We expect the same result when we also examine Individual Therapy.

reintroduction to factorial designs (re-)introduction to factorial designs 5 patterns of factorial...

Documents

w anxiety general anxiety

w anxiety social anxiety

bsimple effects of type

comparison of simple

ividentifying main effects

effects ivs

simple effects arehard

ivmain effects