Experimental Research DesignCausality & Validity

• Threats to Validity– Construct (particular to experiments)– Internal– External – already discussed

Threats to Construct Validity• Mono Method bias

– Remove common method variance– Establish convergent & discriminant

validity• Hypothesis guessing• Inappropriate level of construct

– Usually lower than what occurs in natural setting

Threats to Internal Validity• Time

– History, Maturation• Measurement

– Testing, Instrumentation• Selection

– Randomization, Statistical regression, Mortality • Contamination

– Diffusion, Compensation• Interaction among threats

Internal Validity• Establish that cause led to the effect

– Identify non-experimental (uncontrolled) factors that affect the dependent variable or the relation b/w the independent and dependent variables

10 Threats to Internal ValidityTime

• History– Factors occurring along with the

independent variable• An event occurring at the time of the

intervention that overshadows (or reduces the effect of iv) and causes changes in dv

– E.g., change in manufacturing material along with training influences performance quality

FIGURE 7.1Copyright © 2003 John Wiley & Sons, Inc. Sekaran/RESEARCH 4E

10 Threats to Internal ValidityTime

• Maturation– Biological or psychological processes or

factors that occur w/the passage of time & affect the dependent variable

• E.g., job experience, socialization etc. affect performance

FIGURE 7.2Copyright © 2003 John Wiley & Sons, Inc. Sekaran/RESEARCH 4E

10 Threats to Internal ValidityMeasurement

• Testing– Studies w/pre-post measures or multiple

measures• Increased or decreased sensitization to post-

test measures• Changes in participants due to measurement

itself • Increased fatigue or resentment • Self-consistency effects

10 Threats to Internal Validity• Instrumentation

– Change in the measures from pre to post test

• Change in observers’ frames of reference • Changes in participants’ frames of reference

(e.g., beta & gamma changes p. 43 Saks)• Change in instrumentation quality (e.g.,

performance quality measurement)

10 Threats to Internal Validity• Instrumentation

– Use of different measurement techniques• Establish expected convergent & discriminant

validity• Importance of using different measures to

overcome percept-percept problem

Threats to Internal ValiditySelection

• Randomization– Randomly assign to experimental

conditions • Only in experimental designs• Matching on key variable if random assignment

not possible

Threats to Internal ValiditySelection

• Randomization– Randomly select from population

• Calculate response rate• Establish comparability of respondents to

population on key features• Stratified (v. cluster?) sampling

10 Threats to Internal Validity Selection

• Statistical Regression – Upon repeat measurement, scores usually

“regress” or “move toward” the mean• Usually when extreme scorers are selected into

the study– Can be classified as ‘error’ variation

10 Threats to Internal Validity Selection

• Mortality– Features of participants who do not

complete the study affect the results– Reduced power in testing effects

10 Threats to Internal ValidityContamination

• Diffusion/imitation of treatments– When people in control groups

inadvertently know about or are affected by intervention and that changes their behavior

10 Threats to Internal ValidityContamination

• Compensatory Effects– Knowledge about intervention and

changes in control group members’ behavior is more purposeful

10 Threats to Internal Validity• Interaction among threats

– When multiple threats operate at the same time obscuring the effect of the IV on the DV

Threats to Internal Validity• Time-related

– History, maturation• Measurement

– Testing, Instrumentation• Selection

– Randomization, Statistical regression, mortality • Contamination

– Diffusion, Compensation• Interaction among threats