Effect of motion on Effect of motion on direction discriminationdirection discrimination

• Small error bars. Error variance is really low.Small error bars. Error variance is really low.

• Scientists of such disciplines can explain Scientists of such disciplines can explain most of their variancemost of their variance

0

0.5

1

1.5

2

2.5

3

3.5

4

Low motion condition High motion condition

Effect of similarity of Effect of similarity of objects on categorizationobjects on categorization

• Larger error bars. Error variance is still low.Larger error bars. Error variance is still low.

• Scientists of such disciplines can explain a Scientists of such disciplines can explain a lot of their variancelot of their variance

Low similarity High similarity0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Effect of proximity on group Effect of proximity on group performanceperformance

• Very large error bars. Error variance is high.Very large error bars. Error variance is high.

• Scientists of such disciplines can explain only a small Scientists of such disciplines can explain only a small percentage of variancepercentage of variance

• Note. Mean remains the sameNote. Mean remains the same

Low Proximity High proximity0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Percentage of variance in Percentage of variance in usability?usability?

LargeLarge

MediumMedium

SmallSmall

VariableVariable

Functions of experimental Functions of experimental DesignDesign

• Reduce the error variance, so that you Reduce the error variance, so that you can get focus on the question that you can get focus on the question that you are interested inare interested in

• Experimental statistics make Experimental statistics make assumptions about the data, assumptions about the data, experimental design helps you keep experimental design helps you keep statistically hones.statistically hones.

First rule of statisticsFirst rule of statistics

•Garbage in, garbage out!Garbage in, garbage out!

The most sophisticated statistical The most sophisticated statistical techniques cannot save you if your techniques cannot save you if your design in flaweddesign in flawed

What is experimental What is experimental designdesign

• The design is the general structure of The design is the general structure of the experiment. the experiment.

• Experimental design and the research Experimental design and the research problem are really two separate problem are really two separate things. things.

• Experiment design is not inevitable, Experiment design is not inevitable, i.e., every research problem can be i.e., every research problem can be investigated in many ways.investigated in many ways.

An example An example

• Research question: Can people create better Research question: Can people create better presentations with MultiPoint than presentations with MultiPoint than PowerPointPowerPoint

• Independent Variable: Presentation tool.Independent Variable: Presentation tool. 2 Levels: MultiPoint and PowerPoint2 Levels: MultiPoint and PowerPoint

• Dependent Variable: Quality of presentation.Dependent Variable: Quality of presentation. Quality defined by independent judgment of Quality defined by independent judgment of

presentationpresentation

Between Subjects DesignsBetween Subjects Designs

• Two group, between subjects designTwo group, between subjects design

• Method: Call two groups into the lab, Method: Call two groups into the lab, randomly assign them to groups. Some randomly assign them to groups. Some use PowerPoint, some use MultiPoint. use PowerPoint, some use MultiPoint.

• Results:Results: PowerpointMultipoint2 43 51 65 36 75 8

Within Subjects DesignsWithin Subjects Designs• One group, within subjects designOne group, within subjects design

• Method: Call one group into the lab, randomly Method: Call one group into the lab, randomly assign them to different orders of using assign them to different orders of using PPoint and MPoint PPoint and MPoint

• Results:Results: PowerpointMultipoint2 43 51 65 36 75 8

Design: AB design

AB DesignAB Design

• Takes care of Fatigue and Learning Takes care of Fatigue and Learning EffectsEffects

• Variations: ABBA designVariations: ABBA design

Group 1 Group2Order 1: MultiPoint PowerPoint

Order 2: PowerPointMultiPoint

• The main function of experimental design The main function of experimental design isis to maximize the effect of systematic variance to maximize the effect of systematic variance

(the independent variable)(the independent variable) to minimize the error variance (often to minimize the error variance (often

individual differences)individual differences)

• Error Variance can be of different kinds: Error Variance can be of different kinds: measurement errormeasurement error random differences between experimental random differences between experimental

groups (do your best to eliminate this)groups (do your best to eliminate this) Individual differences within group (not much Individual differences within group (not much

you can do)you can do)

Review: The Design of Review: The Design of ExperimentsExperiments

• Sources of variances in an Sources of variances in an experimentexperiment

• NotationNotationExperimental ManipulationSystematic Variance

Error Variance: Due to Individual Differences

Error Variance:Due to random factors

Between Subject Design: Between Subject Design: Sources of varianceSources of variance

Experimental Manipulation:

Individual Differences between users

Random Variance

Within subject designWithin subject design

Experimental Manipulation:Effect of device

Random Variance

Methods to control Error Methods to control Error VarianceVariance

• RandomizationRandomization

• EliminationElimination

• MatchingMatching

• Additional Independent VariableAdditional Independent Variable

• Statistical ControlStatistical Control

Most of these methods address individual differences variance

RandomizationRandomization• Most effective way to control error variance. If Most effective way to control error variance. If

thorough randomization has been achieved thorough randomization has been achieved then experimental groups can be considered then experimental groups can be considered equal in every way, and you are justified in equal in every way, and you are justified in comparing them. comparing them.

  

• Random SelectionRandom Selection: random selection of units : random selection of units from the populationfrom the population

• Random AssignmentRandom Assignment: Every experimental unit : Every experimental unit (subject or material) has an equal chance of (subject or material) has an equal chance of being in every condition.being in every condition.

Random Assignment with Random Assignment with controlcontrol

• Sometime you cannot totally Sometime you cannot totally randomly assignrandomly assign

• Example: you need to control for Example: you need to control for gender in the PPoint / MPoint studygender in the PPoint / MPoint study Randomly assign from within the Randomly assign from within the

gender groups to the Multipoint / gender groups to the Multipoint / PowerPoint groupsPowerPoint groups

Elimination or ConstancyElimination or Constancy• Sometimes you can control a possible confounding Sometimes you can control a possible confounding

by eliminating it (or eliminating its effect). by eliminating it (or eliminating its effect). Choose the subjects so that they are equal on the Choose the subjects so that they are equal on the

confounding variable.confounding variable.

• Example: PPoint and MPoint experimentExample: PPoint and MPoint experiment Important confounding variable: previous experience in Important confounding variable: previous experience in

using Powerpoint. using Powerpoint. Solution: You could decide that you are only interested in Solution: You could decide that you are only interested in

people who have done between ten and 20 PPoint people who have done between ten and 20 PPoint presentations before and eliminate everyone else from the presentations before and eliminate everyone else from the sample.sample.

Elimination contd.Elimination contd.

• By reducing the difference between By reducing the difference between participants, we reduce the size of participants, we reduce the size of random variance and increase random variance and increase chances of finding meaningful chances of finding meaningful differences.differences.

• Problems with method: Loss of Problems with method: Loss of generalizabilitygeneralizability

MatchingMatching• Match subjects to variable which is substantially Match subjects to variable which is substantially

related to the dependent variable. related to the dependent variable.

• Example: Previous multimedia experience can act Example: Previous multimedia experience can act as a confounding variable in the study. as a confounding variable in the study. People who have multimedia experience will tend to do People who have multimedia experience will tend to do

better presentations, and use Mpoint more effectivelybetter presentations, and use Mpoint more effectively Solution: Match subjects. Recruit 10 subjects, get a Solution: Match subjects. Recruit 10 subjects, get a

baseline measure of their multimedia skills. Try to make baseline measure of their multimedia skills. Try to make sure that mean multimedia skill in equal in both groups. sure that mean multimedia skill in equal in both groups.

Problems with Matching: reduces availability of subjectsProblems with Matching: reduces availability of subjects

Note. The above example pertains to a between subjects design

Confounding variables that Confounding variables that act differently at different act differently at different

levels of the IVlevels of the IV

• Pose a much bigger problemPose a much bigger problem

• Example: Example: Multimedia experience will benefit Multimedia experience will benefit

MPoint usage more than PPoint usageMPoint usage more than PPoint usage

Such a variable can lead to statistically Such a variable can lead to statistically uninterpretable resultsuninterpretable results

Additional Independent Additional Independent VariableVariable

• If the confounding variable actually interests If the confounding variable actually interests you, you can add it into your designyou, you can add it into your design

• Example: Can people with Multimedia Example: Can people with Multimedia experience use Mpoint better than people experience use Mpoint better than people without multimedia experience.without multimedia experience. 22ndnd Independent Variable: Previous experience Independent Variable: Previous experience

with Multimedia.with Multimedia.

Design: 2 x 2 design.Design: 2 x 2 design.

Usability of a Palmtop Usability of a Palmtop DeviceDevice

• Possible Relevant FactorsPossible Relevant Factors Screen sizeScreen size

Screen resolutionScreen resolution

Screen colorScreen color

Hardware configurationHardware configuration