internal validity in research/dissertation
TRANSCRIPT
10/2/2011
1
Krishna Bista
10/04/2011
WHAT IS INTERNAL VALIDITY?
“Internal validity is a crucial measure in quantitative studies, where it ensures that a researcher’s experiment design closely follows the principle of cause and effect” (Shuttleworth, 2009)
Could there be an alternative cause, or causes, that explain my observations and results?”
2
SmokingLow-income groups
� Internal validity: Are the methods correct and the
results accurate?◦ Do the research conditions warrant the conclusions?
◦ Without internal validity results are un-interpretable.
� External validity: Are the findings generalizable beyond that
particular study?◦ To what extent can the results be generalized?
◦ To what populations, settings, treatment variables, and measurement variables?
(Lunenburg & Irby, 2008)
3
10/2/2011
2
Do tests really measure what student learning?
Do college GPAs accurately predict on the job success?
Content validity: CSEQ should measure college experience, not
depression!
Face validity: African Americans take a history test, but all of the
questions were written by Asians.
Criterion validity: Do a person’s reported income predict his or her credit score?
Construct validity: E.g. scores on a “marital satisfaction” scale should be negatively related to spouse abuse.
Predictive validity: Does the Suicide Probability Scale accurately
predict which adolescents are likely to attempt suicide?
4
� Cause and Effect
� Single-Group Design (variables)
� Multiple-Group Design (variables)
5
Temporal precedence
Covariation of cause and effect
No alternative explanations
Cause Effectthen
Time
if X, then Yif not X, then not Y
Program OutcomeCauses?
Alternativecause Alternative
cause
Alternativecause
Alternativecause
Adopted from Brown (2010)6
10/2/2011
3
Administerprogram
Measureoutcomes
X O
Measurebaseline
O
Alternativeexplanations
Alternativeexplanations
Adopted from Brown (2010) 7
O =Observation/MeasurementX= Treatment Intervention
Administerprogram
Measureoutcomes
Measurebaseline
Alternativeexplanations
Alternativeexplanations
X OO
OODo notadministerprogram
Measureoutcomes
Measurebaseline
Adopted from Brown (2010) 8
� History
� Maturation
� Testing
� Instrumentation
� Statistical regression
� Selection
� Research mortality
� Interactions w/ selection
(Brown, 2010; Krathwohl & Smith, 2005; Rudestam, & Newton, 2001)
9
10/2/2011
4
� Specific events (9/11,Tsumami, bomb, flooded
schools, flu), during the life of the experiment
� The longer the interval between the pretest and
posttest, the more viable this threat.
� Example: A Longitudinal Study of the Adaptation of
International Students in the United States. (Toni Falbo,
University of Texas at Austin, 2005)
10
� Changes in physical, intellectual, or emotional
characteristics of the participants
� In longitudinal studies (pre-test/pos-ttest/multi-test), for
instance, individuals grow older, become more
sophisticated, more/less motivated, become
bored/anxious
� Example: Asian Students’ Voices: An Empirical Study of
Asian Students' Learning Experiences at a New Zealand
University (Campbell & Li, 2007).
Changes in language ability, communication skills, and cross-cultural adaptations
11
� Also called “pretest sensitization,” this refers to the
effects of taking a test upon performance on a second
testing.
� Testing becomes a more viable threat to internal
validity as the time between pretest and posttest is
shortened.
� Example: Pre/Post/multi-6th grader -10th grader
Does the test measure factual information that can be
easily recalled or sth else?
12
10/2/2011
5
� Changes in the way a test or other measuring
instrument is calibrated that could account for results of
a research study (e.g. Missouri Test).
� This threat typically arises from unreliability in the
measuring instrument.
� Observer drift—being bored
� Poor survey construction
13
� Occurs when individuals are selected for an
intervention or treatment on the basis of extreme scores
on a pretest.
� Extreme scores are more likely to reflect larger
(positive or negative) errors in measurement.
� Results move towards the mean
14
� This can occur when intact groups are compared. The
groups may have been different to begin with.
� E.g. Does watching American Idol increase singing in
the shower?
15
10/2/2011
6
� Occurs when differential selection is confounded with
maturational effects.
� Example: dating behavior--girls who attended a dance
school and who did not attend
Dance/Nondance Group × Pubertal Status interaction was found.
Breast Development × Dance/Nondance Group interaction was found.
(Gargiulo, Attie, Brooks-Gunn, &Warren, 1987).
16
� The differential loss of individuals from treatment
and/or comparison groups.
� This is often a problem when research participants are
volunteers.
� Volunteers may drop out if they find it is time
consuming.
� Example: A Longitudinal Study of the Adaptation of
International Students in the United States. (Toni Falbo,
University of Texas at Austin, 2005)
17
Whenever measuring affects DV score
� Behavior and self-report (faking)/Judgeaphobia
� May divert attention from experiment instructions to
others
� As an example, three different classrooms—students
with how, low, and average skills (CSEQ—ESL vs. Graduate
Students)
18
10/2/2011
7
� Hide your identity (don’t call yourself a
psychologist/researcher)
� Be informal, friendly, put them at ease
� Control with self-report measures (peer-evaluation,
anonymity, confidentiality/bogus pipeline)(Rudestam & Newton, 2001)
19
Campbell, J., & Li, M. (2007). Asian students’ voices: An empirical study of Asian students' learning experiences at a New Zealand university. Journal of Studies in International Education, 12 (4), 375-396. doi:10.1177/1028315307299422
Brown, T. T. (2010). Research Design. HRH Labor Markets Course: Analyze and Plan Human Resources for Health Global Health Workforce Economic Network, University of California, Berkeley.
Gargiulo, J., Attie, I., Brooks-Gunn, J., &Warren, M. (1987). Girls' dating behavior as a function of social context and maturation. Developmental Psychology, 23 (5), 730-737
Lunenburg, F. C., & Irby, B. J. (2008). Writing a successful thesis or dissertation: Tips and strategies for students in the social and behavioral sciences. Thousand Oaks, California: Sage Publications, Inc.
Rudestam, K. E., & Newton, R. R. (2001). Surviving your dissertation: A comprehensive guide to content and process (2nd Ed.). Thousand Oaks, California: Sage Publications, Inc.
Shuttleworth, M. (2009). Internal Validity. Retrieved from Experiment Resources: http://www.experiment-resources.com/internal-validity.html
20