construct validity and measurement do they measure what they say they measure?
TRANSCRIPT
CONSTRUCT VALIDITY AND MEASUREMENTDo they measure what they say they measure?
Construct ValidityTh
eory
Obs
erva
tion
CauseConstruct
EffectConstruct
Program ObservationsWhat you do What you see
What you think What you think
operationalization
Construct ValidityTh
eory
Obs
erva
tion
CauseConstruct
EffectConstruct
Program ObservationsWhat you do What you see
What you think What you think
Can we generalize back to the constructs?
Construct Validity
the degree to which inferences can legitimately be made from the operationalizations in a study to the theoretical constructs on which they are based
or Did the study really measure what it claimed to
measure?
Central Questions
“Is your operationalization an accurate translation of the construct?”
“Are you measuring what you intended to measure?” “Does your program/treatment accurately reflect
what you intended?”
Construct Validity
translation validity face validity content validity
criterion-related validity predictive validity concurrent validity convergent validity discriminant validity
Important Point•all aspects or elements of construct validity•think of these as things to look for when evaluating construct validity
Translation Validity vs. Criterion-Related Validity
translation validity… assesses whether the operationalization is a good
reflection of the construct criterion validity…
assesses whether the operationalization behaves in the way it should given your theory of the construct (uses other measures (criteria) to assess construct validity)
Translation Validity
Focuses on whether the operationalization is a good reflection of the construct Face Validity: On its face, does the operationalization
look like a good translation of the construct?
Translation Validity: Face validity
Construct• satisfaction with KNR 164 coursePossible Operationalizations (ways to measure construct)1. 2. 3. 4.
Question:Which of these are more or less reasonable on the
face of it?
Translation Validity: Face validity
Construct• FitnessPossible Operationalizations (ways to measure construct)1. 2. 3. 4.
Question:Which of these are more or less reasonable on the
face of it?
Translation Validity: Content Validity
Content Validity: Operationalization is checked against the relevant content domain for the construct
Often involves researching just how the construct is defined by those in a position to know (experts)
Translation Validity: Content Validity
Construct fitness programPossible Operationalization (key elements?)
Question:Are these the correct elements of the construct?
Criterion Validity
The performance of your operationalization (i.e., measure) is checked against a criterion.
A prediction of how the operationalization will perform on some other measure based on your theory or construct
“Validating” a measure based on its relationship to another measure
Criterion Validity
Predictive Validity: Operationalization’s ability to predict something it should theoretically be able to predict e.g.: GRE and grad school performance (!)
Concurrent Validity: Operationalization’s ability to distinguish between groups which theoretically should be different e.g.: fatness test for athletes and non-athletes
Criterion Validity
Convergent Validity: Degree to which the operationalization is similar to (converges on) other operationalizations to which it theoretically should be similar e.g.:
Discriminant Validity: Degree to which the operationalization is not similar to other operationalizations to which it theoretically should not be similar e.g.:
Threats to Construct Validity
inadequate preoperational explication mono-operation bias mono-method bias interaction of different treatments interaction of testing and treatment restricted generalizability across constructs confounding constructs and levels social threats to construct validity
Examples of Threats to Construct Validity
construct not defined clearly enough only one possible example of the construct (either
IV or DV)
Examples of Threats to Construct Validity
inaccurate labeling of construct missing important elements failure to define or consider “dose”
social issues participants guessing what they are “supposed” to do
or say participants being apprehension experimenter’s expectancies biasing observations
being made
Below is a research problem. Identify which of the threats to construct validity may be of major concern.
General idea behind the research scenario (a quotation from our researcher): “I feel that plyometric strength training is more effective for gaining strength than isometric strength training. I’ve done plyometrics for
years, and it has worked wonders.”
An undergrad class taught by the researcher is split into 3 groups of 30. One third is assigned to a plyometric strength-training program, 1/3 to an isometric program, and 1/3 do nothing. Before assigning them, the researcher makes sure to tell the entire class about the purpose of the research, and explains we are doing it to see if the researcher’s suspicions about plyometrics are correct.
Before and at the end of the programs, all students are tested on a measure of strength - a grip dynamometer. This test is supervised by the researcher to make sure proper procedures are followed.
It is expected that the plyometric group will make the greater strength gains.
Guiding Questions
Construct Validity –Measures/Observations
1. What, in theory, are the researchers trying to assess or measure (list each construct)
Answer the following questions for each construct included in the study.
2. Do the researchers explicitly define the construct? If so, how?
3. If the construct is not explicitly defined by the authors, what does the construct mean to you (in theory)?
4. How did the researchers operationalize the construct? That is, how exactly did they measure/assess the construct?
5. In your opinion, is the operationalization of the construct a reasonable approximation of the theoretical construct? In other words, does the measure they used in the study match up with what they said they were trying to measures? [This is the key Construct Validity question]
6. Do you see any limitation with their operationalization (i.e., are any of the common threats an issue)?
7. Do the researchers provide or present any evidence of content validity or criterion-related validity?
Guiding Questions
Construct Validity – Interventions/Treatments
1. What, in theory, are the researchers trying to test as their intervention or treatment?
2. Do the researchers provide some idea of what their intervention/treatment should look like in theory? If not, what should the intervention/treatment be to you (in theory)?
3. How did the researchers operationalize the intervention/treatment? That is, what exactly did they have the subjects or groups of subjects do?
4. In your opinion, is the operationalization of the intervention/treatment a reasonable approximation of what the research wanted to test in theory? In other words, does the intervention/treatment they used in the study match up with what they said they were trying to do? [This is the key Construct Validity question]
5. Do you see any limitation with their operationalization (i.e., are any of the common threats an issue)?
Overall (final) question in each case:
Do the limitations to construct validity: A. change the meaning of the study’s conclusions or B. have the potential to alter the results of the study.
Practice Activity #1: Evaluate the construct validity of this studyGeneral idea behind the research scenario (a quotation from our researcher):
“I feel that plyometric strength training is more effective for gaining strength than isometric strength training. I’ve done plyometrics for years, and it has worked wonders.”
An undergrad class taught by the researcher is split into 3 groups of 30. One third is assigned to a plyometric strength-training program, 1/3 to an isometric program, and 1/3 do nothing. Before assigning them, the researcher makes sure to tell the entire class about the purpose of the research, and explains we are doing it to see if the researcher’s suspicions about plyometrics are correct.
Before and at the end of the programs, all students perform a 1RM leg extension test as a measure of strength. This test is supervised by the researcher to make sure proper procedures are followed.
During the 4 weeks of training, the subjects in the plyometric group did 10 drop jumps each day from a height of 2 feet, while the subjects in the isometric group performed 3 sets of 10 reps of the following exercises using Nautilus equipment: bench press, shoulder press, and biceps curls. Those in the control group were instructed to do no physical activity during the 4 weeks.
It was expected that the plyometric group will make the greatest strength gains.
Practice Activity #2
Use the guiding questions to evaluate the construct validity of measures used in the study distributed in class