6. methods of experimental...

Chapter 6: Control Problems in Experimental Research

6. Methods of Experimental Control

1

• Understand:

• Advantages/disadvantages of within- and between-subjects experimental designs

• Methods of controlling for group differences in between-subjects experimental designs.

• Counter-balancing techniques for controlling sequence effects in within-subjects experimental designs

Goals

2

Population

Condition 1 Condition 2

Sample

Sampling

Group 1 Group 2

Random Assignment

Between-Subjects Experimental Design(single factor, 2 levels)

3

Within-Subjects Experimental Designs (single factor, 2 levels)

4

Condition 1 Condition 2

Population

Sample

Sampling

Sample Sample

1/2

Sample Sample

1/2

• Different groups of people assigned to each level of IV

• Requires more participants, but avoids sequence effects (e.g., practice or fatigue).

• Potential Validity Problem: Are the people in the groups the same to begin with?

• If not, group differences = a confound

Between subjects designs

5

• Technique for minimizing group differences.

• Works simultaneously on ALL variables that might lead to group differences. Very powerful.

• Simply divide P’s among levels of IV in a random (not arbitrary!) way.

• NOT to be confused with random sampling!

Simple Random Assignment

6

An Aside: Randomness

• Random is not the same as arbitrary.

• Randomness can be thought of as “systematically non-systematic”. That is, you set up a procedure to eliminate any possible biases.

• Arbitrary procedures, such as deciding haphazardly who goes in which group, may contain unknown biases

7

An Aside: Randomness• Two flavours of randomness:

• Random with replacement: All options are there on every trial (dice, coin tosses)

• Random without replacement: When an option is picked on a given trial, it is no longer available for later ones (cards)

• Use the latter with random sampling and random assignment

8

• Typically, subjects are “shuffled” randomly using computer-generated random numbers.

• Physical mixing can also be used.

• Careful! Must use a method that is “random without replacement”. Example: drawing cards from a deck without putting them back in the deck for the next P. Counter-Example: Flipping a coin is right out! That is “random with replacement”

Simple Random Assignment

9

• List all levels of IV in column A, with n repeats, where n is the # of individuals who will be in each group.

• Create a list of random numbers, using =rand() in column B, then sort according to column B (cut and paste values)

• If all participants are known ahead of time, just paste the list into column C

• If participants are not known ahead of time, test them as they come in, in the order of the list.

Random Assignment in Excel: One IV

10

• List all levels of IV1 in column A, with n repeats, where n is the # of individuals who will be in each level of IV1.

• List all levels of IV2 in column B, with n repeats within each level of IV1, where n is the number of individuals in each condition.

• Rest is as for one IV.

Random Assignment in Excel: Two IVs

11

Discussion / Questions

12

• Technique used in between-subjects designs to avoid “clumping” of conditions at particular times.

• (Also used in within-subjects designs, but more on this later... )

• In sequential testings, simple RA may create a confound: Example: One might end up doing most of Level 1 before any of Level 2.

• Whether such clumping is a problem depends on the likelihood of history confounds, and the size of the sample, but it’s never a bad idea to avoid it.

Block Random Assignment

13

• In block RA, the set of all conditions is shuffled several times, and a series of shuffled sets of conditions is created Example: Experiment with three conditions might produce a sequence like3 1 2 1 3 2 2 1 3 3 2 1 1 3 2 . . . Counter-example: With simple RA, same experiment might produce a sequence like3 3 1 1 3 2 3 3 1 2 2 1 1 2 2 . . .

• Note that, when using block RA, number of participants should ideally be an even multiple of the number of conditions

Block Random Assignment

14

• E.g., Four conditions with n=10 each.

• In simple RA: Shuffle 10♥, 10♦, 10♣, 10♠

• But might (just by chance) end up drawing most of the ♥’s before any ♣’s are drawn (for example).

• Block RA: Instead create 10 “block decks” of four cards each: 1♥, 1♦, 1♣, 1♠ in each deck.

• Shuffle each block deck, then stack all the block decks on top of one-another.

Example of Block RA

15

• With simple randomization, you might end up with a sequence like this:

• But with block randomization, you end up with a sequence like so:

Block Random Assignment

♥ ♥ ♦ ♦ ♥ ♥ ♥ ♠ ♣ ♥ ♣ ♦ ♠ ♦ ♥ ♠ ♠ ♥ ♠ ♥ ♦ ♠ ♥ ♣ ♠ ♦ ♠ ♣ ♦ ♣ ♣ ♣ ♦ ♣ ♠ ♦ ♠ ♦ ♣ ♣

♥♣♦♠ ♠♦♣♥ ♣♥♦♠ ♣♥♦♠ ♥♣♠♦ ♣♠♦♥ ♥♠♦♣ ♦♣♠♥ ♠♦♣♥ ♣♠♥♦

Start of Study Middle of Study End of Study

Start of Study Middle of Study End of Study

16

• RA works well with large N. What is “large”? Ideally 30, but as little as 10 is acceptable for small studies.

• But, chance of non-equivalent groups rises as N drops.

• What to do if you’re stuck with small N?

Random Assignment & Number of Participants

17

• Instead of random assignment, test participants on matching variable(s)

• Then assign P’s to groups such that groups have equal means (or frequency distributions if MVs are nominal) on the matching variable(s)

• Must have theoretical reason to expect an effect of matching variable

• Matching variable must be testable practically and without introducing testing effects.

Matching

18

• Easiest to do with variabless that can be assessed without lengthy testing Examples: Age, Gender, Weight...

• How many factors to match on? Can get complicated. May result in having to turn participants away if no match can be made

• May be simpler to test more subjects and let random assignment do its magic.

Matching

19

GPA

9.1

9.0

8.5

8.0

7.3

7.1

7.0

6.8

6.5

5.5

9.1 9.0

8.5 8.0

7.3 7.1

7.0 6.8

6.5 5.5

Step 1: Order Values

Step 2: Create pairs of adjacent values

Step 3: From each pair, randomly assign one to each group

Group 1 Group 2

9.0 9.1

8.5 8.0

7.3 7.1

6.8 7.0

5.5 6.5

µ = 7.42 µ = 7.5420

Discussion / Questions

21

Instructions:

In a moment I’m going to show you a video. It shows 6 people playing basketball. I want you to watch the video and keep a silent mental count of the number of passes between players. But to make it a little more difficult, I want you to keep two separate counts, one for the number of passes through the air, and another for number of bounce-passes, that is, the number of times they bounce the ball to one another.

If you’re on the right side of the class (your right, my left), do this for the white-shirted players only.

If you’re on the left side of the class (your left, my right), do this for the black-shirted players only.

When we’re done, I’ll ask you to write down the two numbers (number of bounce passes and number of air passes) and give the data to me.

Remember, just keep a silent count, don’t make any noise or marks with your pen or anything like that.

A Between-Subjects Experiment

22

23

What to Take Away From This?

• Perception: You don’t actually see what’s out there, just a reconstruction

• Cognition: There are limits to human attentional load

• Phil. of Science: It pays to observe the same thing several times, sometimes looking at details (=experiment), sometimes looking at the “big picture” (= naturalistic observation)

• RM&E: Some things can’t be repeated within-subjects

24

Within-Subjects Experimental Designs

Condition 1 Condition 2

Population

Sample

Sampling

Sample Sample

1/2

Sample Sample

1/2

25

• a.k.a. “Repeated measures designs”

• Same group goes through all levels of the IV.

• Often used when time to do one condition is small, or when available population is small.

• Fewer participants needed, no group effects, but may be impractical for some tasks.

Within-Subjects Designs

26

• Allows more statistical power

• More participants per condition for a given grand N

• Don’t have to deal with between-groups variance (even with RA, there’s always some difference between groups that can obscure experimental effects)

• BUT, must be careful of sequence effects

Within-subjects Designs

27

• Going through level A of the IV may affect performance on level B.

• Progressive Sequence Effects:

• Practice effect: Participant gains knowledge, warms up, focuses, etc.

• Fatigue effect: Participant gets tired, bored, overwhelmed, etc.

Sequence Effects

28

• Carry-over effects: Non-symmetrical sequence effects. Doing Level A then Level B not the same as doing Level B then Level A.

• Common when levels vary in difficulty: “simple then hard” is easier than “hard then simple”.

• In this case, best to switch to between-subjects design.

Sequence Effects

29

Counterbalancing

• Group of techniques for minimizing progressive sequence effects in within-subjects experiments

• Complete counterbalancing

• Partial counterbalancing

• Sequence randomization

• Sequence randomization with constraints

• Latin square

• General idea is to equalize the number of participants who do each level in each order

30

• Equal number of participants goes through each possible order of conditions Example: With two condition, half of Ps do 1-then-2, other half do 2-then-1

• The ultimate form of counterbalancing, but not always practical.

• Number of orders of levels is “n factorial” or “n!”, where n is number of levels.

CompleteCounterbalancing

31

Factorial

# of Levels

# of Orders

2 2

3 6

4 24

5 120

6 720

N! = N × N-1 × N-2 ... × 1

32

Factorial

• Why is it N! ?

• Consider a case where you have 4 chairs and need to seat 4 people. How many people can you choose from to go in the... ...1st chair? 4...2nd chair? 3 (because one is in the 1st)...3rd chair? 2 (other 2 already seated)...4th chair? 1

33

Discussion / Questions

34

• Sequence randomization: Necessary when large number of levels, but adds noise.

• Sequence randomization with constraints:

• Fellows Numbers: Ensure that correct answer is not the same more than X times in a row

• Same stimulus does not appear on sequential trials

Partial Counterbalancing

35

• From an ancient roman game: Given an X by X grid, and X different symbols, can you place the symbols in the grid so that each appears only once per row and once per column? Similar to Sudoku puzzles.

• Even harder: Can you make it so that each symbol appears directly to the right of each other symbol once and only once? (only possible when X is even). This is called a balanced latin square

Latin Square

36

• A completed Balanced Latin Square can be used as a form of partial counter-balancing

• Each participant runs through the conditions in the order indicated by one row of the BLS

• Number of participants must be evenly divisible by number of levels

Latin Square

37

1 2 3 4 5 6

1 A B F C E D

2 B C A D F E

3 C D B E A F

4 D E C F B A

5 E F D A C B

6 F A E B D C

1 2 3 4 5 6

1 A B F C E D

2 B C A D F E

3 C D B E A F

4 D E C F B A

5 E F D A C B

6 F A E B D C

1 2 3 4 5 6

1 A B F C E D

2 B C A D F E

3 C D B E A F

4 D E C F B A

5 E F D A C B

6 F A E B D C

6x6 Balanced Latin SquareSequential Position

Ord

er #

38

Latin Square Design

• If you run one participant through each of the 6 orders, then:

• Each of the 6 levels will have been done once in each of the 6 possible sequential positions.

• Each of the 6 levels will have been immediately preceded by each of the other 5 levels once and only once.

• If you run 60 people, then 10 will have gone through each order

39

Latin Square Design

• For example, if you test 60 people (6 x 10), then:

• 10 will have done A first, 10 B first, 10 C first...

• 10 will have done A second, 10 B second, ...

• 10 will have done level A immediately preceded by B, 10 will have done A immediately preceded by C, etc...

40

Creating a Balanced Latin Square (if X is even)

• Build the first row according to the pattern: A B (x) C (x-1) D (x-2) E (x-3) etc... where x is the highest letter you’re using (e.g., F if doing a 6×6). With 6 levels, row 1 is:

A B F C E D

• Build the remaining rows by incrementing the letters by 1 (i.e., A becomes B, B becomes C...). Row 2 is

B C A D F E

Note that we “wrap back” to A when incrementing F

41

Latin Square With Odd Number of Levels

• Previous system only works with even # of levels. A 5×5 or 7×7 latin square cannot be balanced.

• With uneven # of levels, create latin square plus a left-right mirror of it.

• Run an equal number of participants through each of the orders in these two latin squares

1 2 3 4 5

1 A B E C D

2 B C A D E

3 C D B E A

4 D E C A B

5 E A D B C

1 2 3 4 5

6 D C E B A

7 E D A C B

8 A E B D C

9 B A C E D

10 C B D A E

Sequential Position

42

Ord

er

Summary: Counterbalancing

• 2-3 levels: Complete counterbalancing

• 4-8 levels: Latin square

• 4+ levels: Sequence randomization, possibly with constraints

43

• What if subjects experience each condition more than once?

• Reverse counterbalancing:

• ABCD-DCBA-ABCD-DCBA...

• Block randomization:

• BADC-CBAD-DCAB-ADCB...

• Block randomization with constraints

Counterbalancing w/ Multiple Exposures

44

• Cross-sectional study

• Between groups: Test 5, 7, 9 years olds

• Faster than following from 5-9

• Problem: Cohort effects.

• Longitudinal study

• Within groups: Follow 5 year olds until 9 years old.

• Takes a long time!

• Problem: Attrition

• Other methods combine the two

Within vs. Between in Developmental Psych

45

• Participant differences: random assignment, block randomization, matching

• Order effects: Full & partial counterbalancing

• Participant bias: Blind procedures. Removal of demand characteristics

• Experimenter bias: Automation, double-blind procedures

• Floor & ceiling effects: Use procedures that are neither too difficult nor too easy.

Summary:Confounds & Controls

46

Think Twice...

• Carpenter’s adage: “Measure twice, cut once”

• Scientist’s adage: “Think twice, measure once”

• Do not rush experimental design, there are many pitfalls to be avoided and careful design will save time in the long run.

47

Discussion / Questions

• What is the difficulty with reverse counterbalancing?

• What method of counterbalancing would you use for an experiment with 5 levels?

48