validity does test measure what it says it does? is the test useful? can a test be reliable, but not...
Post on 20-Dec-2015
217 views
TRANSCRIPT
![Page 1: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/1.jpg)
Validity
• Does test measure what it says it does?
• Is the test useful?
• Can a test be reliable, but not valid?
• Can a test be valid, but not reliable?
![Page 2: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/2.jpg)
Types of validity
• Face validity– Important only so far as it doesn’t interfere with
an examinee’s willingness to cooperate.
• Content validity– How well does the test cover areas of content
that it should?– How adequately does it sample the universe of
behavior it was designed to assess?
![Page 3: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/3.jpg)
Content validity (cont.)
• Panel of “experts”– Is the item/content essential?– Lawshe (1975) >50% of experts see skill as
essential
• Important for: – Achievement/classroom tests– Training program exams– Professional exams
![Page 4: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/4.jpg)
Criterion-Related Validity
• How well does a test score relate to another score/variable of interest?– Correlate test with criterion
• Standard against which test is evaluated
• Concurrent
• Predictive
![Page 5: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/5.jpg)
Criterion-Related Validity (cont.)
• Criterion should be– Reliable
• Reliability limits validity; can’t be valid if not reliable.
– Relevant
– Valid
– Uncontaminated• Criterion measure has been based in part on predictor measure
![Page 6: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/6.jpg)
Criterion-Related Validity (cont.)
• Concurrent validity– Criterion immediately available– Present standing on a criterion
• Diagnosis, score on another test
– Used to predict the performance of new test takers or for people for whom the criterion isn’t available.
![Page 7: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/7.jpg)
Criterion-Related Validity (cont.)
• Predictive validity– Test given, criterion measured later– Ex. ACT & College GPA; employment test &
job performance
• Incremental validity
![Page 8: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/8.jpg)
Base Rate & Decision Theory
• Base rate: proportion of population who possess a certain trait, characteristic or attribute– % of EIU undergrads who graduate– % of African Americans with sickle cell anemia
• Base rate affects usefulness of tests
![Page 9: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/9.jpg)
Decision Theory
• 4 outcomes
False rejections/negatives
Valid Acceptances/
Positives
Valid Rejections/
negatives
False Acceptances/
Positives
![Page 10: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/10.jpg)
Cut scores & Hit rates
False rejections/negatives Valid Acceptances/
Positives
Valid Rejections/
negatives
False Acceptances/
Positives
![Page 11: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/11.jpg)
Cut scores & Hit rates (cont.)
• Reciprocal relationship between # of false rejections and # of false acceptances
• Which is more acceptable: to limit the number accepted who shouldn’t be, or to minimize the # rejected who could be successful?
![Page 12: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/12.jpg)
Construct Validity
• Construct:– Scientific idea hypothesized to explain behavior
– Postulated attribute of people, assumed to be reflected in test score
– Ex.: intelligence, self-esteem, motivation
• Construct validity: Does the test measure the construct?– Gives theoretical meaning to scores;
– Subsumes all other types of validity
![Page 13: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/13.jpg)
Construct Validity (cont.)
• Convergent evidence/validity
• Divergent/discriminant evidence
• Factor analysis– Data reduction/simplification of complex
correlational matrices … to reveal major dimensions that underlie a set of items
– A factor is considered to be the construct that best represents relationships among variables
![Page 14: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/14.jpg)
Factor Analysis (cont.)
• Methods of factor analysis– Exploratory
1. Correlation matrix
2. Factor matrix with loadings
3. Label factors
• Used to develop or eliminate items or scales from composite scores
![Page 15: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/15.jpg)
Factor Analysis (cont.)
• Confirmatory factor analysis– Goodness of fit– After test has been developed
![Page 16: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/16.jpg)
Validity & Bias
• Bias: a factor inherent within a test that systematically prevents accurate, impartial measurement– Bias implies systematic, not random variation
• Can you make equally valid predictions for different groups?
![Page 17: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/17.jpg)
Bias in Predictions
• Questions of regression– Slope– Intercept– Error of estimate
![Page 18: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/18.jpg)
Slope Bias
Bias & the DAS
60
80
100
120
140
75 85 100 115
General Conceptual Ability Scores
Word Reading Scores
Whites
Asian Americans
Linear (Whites)
Linear (AsianAmericans)
![Page 19: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/19.jpg)
Intercept Bias
Bias & the DAS
0
20
40
60
80
100
120
140
1 2 3 4 5 6
General Conceptual Ability Scores
Basic Number Skills
Series1
Series2
![Page 20: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/20.jpg)
Rating error
• Leniency Error
• Severity Error
• Central Tendency Error
• Halo Effect
![Page 21: Validity Does test measure what it says it does? Is the test useful? Can a test be reliable, but not valid? Can a test be valid, but not reliable?](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649d485503460f94a23cf4/html5/thumbnails/21.jpg)
Test Fairness
• Is the test used in an impartial, just, and equitable manner?
• Good tests Discriminate among individuals– Are group differences due to inadequate tests?– Is the test being used fairly?