research problems. hawthorne effect u western electric’s hawthorne plant 1939 study of light...
Post on 19-Dec-2015
217 views
TRANSCRIPT
Hawthorne Effect
Western Electric’s Hawthorne Plant 1939 study of light intensity
productivity went…up Potential Solutions:
run experiment for a longer perioduse a control group
John Henry Effect
legend of black railway worker control group overcompensates Potential Solutions:
don’t do threatening experiments don’t set up obviously competitive situations don’t tell control group that they are control group
• conduct in another school somewhere else unfortunately, produces new variable of different
school, neighbourhood, etc.!
Placebo Effect
introduce placebo in attempt to make conditions in treatment and control groups identical
placebo effects are reactions that (after taking the placebo) cannot be explained by the chemical or medical effects of the placebo.—psychological factors
Placebo Effect Potential Solutions double-blind experiment
secrecy but then violate principle of informed consent
screen out or balance number of placebo reactors in treatment & control groups
the Pygmalion effect
Rosenthal and Jacobson, 1968 self-fulfilling prophecy Potential solution:
do not tell subjects what experiment is about but what happens to principle of informed consent?
Demand Characteristics
rumour setting instructions status and personality of researcher unintentional cues from experimenter experimental procedure itself
Demand Characteristics
Potential Solutions
reduce clarity of demand characteristics
generate alternative demand characteristics
reduce subjects’ motivation to respond to demand characteristics
Validity
Internal Validity– control experiment to eliminate
extraneous variables External Validity
– outcome can be generalized to other populations in other settings
Threats to Internal Validity History
change producing events in addition to the experimental treatment
Maturationsubjects grow older, learn more, can do
more
Threats to Internal Validity
Testingin pre-test/post-test designs — subjects may
remember questions from pre-test
Instrumentationchanges in measurementchanges in observerchanges in subjects (testing effects)
Threats to Internal Validity
Differential SelectionDifferences between treatment and control
groupE.g., volunteers vs non-volunteers
Experimental Mortality (or Attrition)subjects that drop out of experiment may
differ from others in important ways
Threats to Internal Validity
Testing & Experimental Treatment InteractionIn pre-test/post-test design, pretest may
interact with experimental treatment to exaggerate result
Statistical Regression
External Validity
1. Population Validity– can results be generalized from specific
sample to the population from which sample was drawn
2. Ecological validity– can results be generalized from contrived
conditions created by experimenter to another set of environmental conditions (i.e., real world)
Experimental Design
Purpose is to design an experiment that controls for as many extraneous variables as possible
– Campbell and Cook, classic 1968 paper categorized experimental designs by how many variables they controlled
The One-Shot Case Design
hardly experiment at all– can’t be sure result was result of treatment, not
history, maturation, etc. no control over group selection could have problem with subject mortality (e.g.,
transient student population) since only tested once, can’t even measure gain —
maybe students knew it before we started consequently, results are largely meaningless
One Group Pre-test/Post-test Design
Uncontrolled Extraneous Factors (all of them!) history e.g., Hawthorne effect maturation testing
Pre-test may have contributed to higher scores on post-test due to greater familiarity with types of questions or focus on certain topics
instrumentation pre-test and post-test the same? Observer the same?
selection: could be an atypical group mortality interaction of testing & experimental treatment statistical regression
One Group Pre-test/Post-test Design useful for studying stable dependent variable justified when
– extraneous factors can be estimated with a high degree of certainty or
– can be safely assumed to be nonexistent• E.g., not maturation because change is too dramatic for
maturation to explain it
Posttest-Only Nonequivalent Groups Design
X O1
-----------------------------------------
O2
(dotted line means not random selection)
Posttest-Only Nonequivalent Groups Design also called “static group comparisons” still a relatively weak design advantage over single one shot design that one can
compare, so cancel out maturation, etc.
Problems: Differential selection experimental mortality
Non-Equivalent Control Group Design
Expt O1 X O2
-----------------------------------------
Cntl O3 O4
can compare average (mean) gain O2-O1
with average gain O4-O3
Non-Equivalent Control Group Design stronger design (weakest of the strong
designs) common design in education because usually
can’t randomize assignment of students to classes
pre-test measures whether initial groups are similar on tested variable– could also match subjects based on pre-test but may miss other factors which could impact
results (e.g., better teacher in one group)
Non-Equivalent Control Group Design Factors controlled by inclusion of a control group
history maturation testing instrumentation (assuming same for both groups) statistical regression
Factors still NOT controlled differential selection experimental mortality (if any) testing and experimental treatment interaction
Pre-test/Post-test Control Group Design still two uncontrolled factors Intersession History
– events that are specific to one group and not the other which occur during the group sessions
– To control for interssion history, need to balance (control) such factors as
• experimenters• time of day• day of week
by randomly assigning to groups of subjects
Pre-test/Post-test Control Group Design Interaction of testing and treatment
– Cannot be avoided when there is a pretest– eliminating the pre-test would therefore
improve design
The Post-test Only Control Group Design do not need pre-test with random selection disadvantages
– random assignment may not be fully successful in eliminating initial differences between control and experimental groups
– cannot form subgroups (i.e., high, medium, & low) to determine whether the experimental treatment has a different effect on subjects at different levels of the variable as measured by the pretest (because no pretest)
Solomon Four-Group Design basically, a pretest and non-
pretest experiment at the same time
controls for everything, except differential mortality