issues in experimental design for the study of atypical language development
DESCRIPTION
TRANSCRIPT
08/10/2013
1
Issues in experimental design for the study of
atypical language development
David Saldaña
DESIGNING YOUR STUDY: CONTROL
08/10/2013
2
Controlled experiments This type of experiment is conducted in a well-controlled environment – not necessarily a laboratory – and therefore accurate measurements are possible.
Easier to replicate
Precise control of extraneous and independent variables.
The artificiality of the setting and lack of generalization
Demand characteristics or experimenter effects may bias the results
McLeod, S. A. (2012).
Field experiments The experimenter still manipulates the independent variable, but in a real-life setting (so cannot really control extraneous variables).
More likely to reflect real life because of it natural setting
Less likelihood of demand characteristics affecting the results
Can be used in situations in which it would be ethically unacceptable to manipulate the independent variable, e.g. researching stress.
Less control over extraneous variables
Less replicable
McLeod, S. A. (2012).
08/10/2013
3
Natural experiments Natural Experiments are conducted in the everyday environment of the participants but here the experimenter has no control over the IV as it occurs naturally in real life.
More likely to reflect real life because of it natural setting
Less likelihood of demand characteristics affecting the results
Can be used in situations in which it would be ethically unacceptable to manipulate the independent variable, e.g. researching stress.
Less likelihood of demand characteristics affecting the results, as participants may not know they are being studied.
No control over extraneous variables
Less replicable
Possible more expensive
McLeod, S. A. (2012).
A perhaps “silly” research question
Does vocabulary influence the comprehension of questions more than the comprehension of imperatives?
08/10/2013
4
A question that is a little more complicated
Do children with SLI understand questions better than typically developing children?
Do children with SLI understand questions better than imperatives?
DESIGNING YOUR STUDY: CONDITIONS
08/10/2013
5
How is your IV distributed?
Independent Measures
Repeated Measures
Matched Pairs
Independent measures (between-group)
• Each group gets one condition
• Different participants in each group • Avoids practice
• More people needed
• Participant variables could affect results.
08/10/2013
6
Task 1
• Measurement
Task 2
• Measurement
Repeated measures (within-group)
- Less participants - Precision determined by variation within same
subject - May be the only design that answers the
questions of interest. - Order effects: practice and fatigue effect
Counterbalancing
Moment 1
A
B
Moment 2
B
A
08/10/2013
7
Back to our “silly” study
Lets design the tasks and the data collection
Does vocabulary influence the comprehension of questions more than the comprehension of imperatives?
A little bit more interesting (perhaps) research question
Does vocabulary influence the comprehension of questions more than the comprehension of imperatives in SLI?
08/10/2013
8
Task 1
• Measurement
Task 2
• Measurement
Mixed-design
Task 1
• Measurement
Task 2
• Measurement
Gro
up
1
Gro
up
2
Some imaginative results
70
77 73 75
90 90 95
99
92 90
100
90
0
20
40
60
80
100
120
IQ Vocabulary ComprehensionQuestions
Word Reading SES ComprehensionImperatives
Stan
dar
d S
core
s
SLI Typical
08/10/2013
9
MATCHING
Matching your groups: why?
70
77 73 75
90 90 95
99
92 90
100
90
0
20
40
60
80
100
120
IQ Vocabulary ComprehensionQuestions
Word Reading SES ComprehensionImperatives
Stan
dar
d S
core
s
SLI Typical
08/10/2013
10
Matching your groups: on what?
Imp.Comp
6 8 10 2 4 6 8
24
68
68
10
IQ
QuestComp
510
15
24
68
SES
2 4 6 8 5 10 15 4 6 8 10
46
810
Voc
Your groups: three group design
• Group of interest
• Control group, matched on chronological age
• Control group, matched on variable of interest
08/10/2013
11
Some issues
• Match on variables of interest to the study
• Match average and distribution
• Study carefully your exclusions
Recruit clinical sample
Decide variables of
matching
Recruit age controls
Recruit “level” controls
Let’s do it! Matching exercise
Some issues
• Match on variables of interest to the study
• Match average and distribution
• Study carefully your exclusions
Recruit clinical sample
Decide variables of
matching
Recruit age controls
Recruit “level” controls
Check and report
08/10/2013
12
What statistical test?
An ideal world
Facon et al. (2011)
What statistical test?
An ideal world
08/10/2013
13
What statistical test?
An ideal world The real world
Non-normal distributions Different range and variance Different distributions
What statistical test?
None: look at the picture first!
Back-to-back histogram Back-to-back stem-and-leaf
08/10/2013
14
What statistical test?
What statistical test?
Mann-Whitney?
Two-sample Kolmogorov-Smirnov test
For small sample sizes: exact version
• Does not assume prior “shape” in distributions
• Tests for the differences in distribution
08/10/2013
15
Matching your groups: how much? The easy-peasy way
Matching your groups: how much?
http://intuitor.com/statistics/T1T2Errors.html
A brief recall of significance testing
Match No Match
I say No Match
I say Match
08/10/2013
16
“Casual acceptance of the null hypothesis” (Harcum, 1990)
< .2 too low
.2 to .5 ambiguous
> .5 fine
Frick (1995)
http://intuitor.com/statistics/T1T2Errors.html
Some things to remember • Population comparisons are group comparisons
(Paradis, 2010)
08/10/2013
17
Some things to remember
• Each group must be there for a reason
(Paradis, 2010)
An example: why each group?
OCS 20 adolescents (13-21) task with sound
OSS 20 adolescents (13-21) task with no sound
NO 20 children (6-11) oral language equivalent to SAL
Two groups of deaf adolescents
SAL 16 deaf (13-21) good oral language
SBL 16 deaf (13-21) poor oral language
Three groups of hearing adolescents
Torres (2013)
08/10/2013
18
Some things to remember
• Check matching while you are recruiting
(Paradis, 2010)
Keep your lab notes up to date
•Some important decisions to record: •Changes and discussions on criteria •Subjects in – subjects out. •Transcriptions and criteria if using MLU
(Paradis, 2010)