psychometric evaluation and calibration plan
DESCRIPTION
Psychometric Evaluation and Calibration Plan. Ron D. Hays, Ph.D. October 14, 2006 ISOQOL (Opala) 2:00-3:30 pm. PROMIS Domains. Physical functioning (Hays/Bjorner) Pain (Revicki/Cook) Fatigue (Lai) Emotional distress (Choi/Reise) Social/role participation (Bode/Hahn). Datasets. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/1.jpg)
1 04/21/23
Psychometric Evaluation Psychometric Evaluation and Calibration Planand Calibration Plan
Ron D. Hays, Ph.D. Ron D. Hays, Ph.D.
October 14, 2006 October 14, 2006
ISOQOL (Opala)ISOQOL (Opala)
2:00-3:30 pm2:00-3:30 pm
![Page 2: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/2.jpg)
2 04/21/23
PROMIS Domains
Physical functioning (Hays/Bjorner)
Pain (Revicki/Cook)
Fatigue (Lai)
Emotional distress (Choi/Reise)
Social/role participation (Bode/Hahn)
![Page 3: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/3.jpg)
3 04/21/23
Datasets
Cancer Item Banks (Northwestern)
Digitalis Investigation Group Study--randomized double-blind placebo-controlled trial evaluating effect of digoxin on mortality in 581 patients with heart failure and sinus rhythm.
IMMPACT--internet-based survey of individuals with chronic pain from the American Chronic Pain Association website
Medical Outcomes Study--observational study of persons with hypertension, diabetes, heart disease, and/or depression in Boston, Chicago, and Los Angeles
WHOQOL-100 data (n = 442 from U.S. field center)
![Page 4: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/4.jpg)
4 04/21/23
Types of AnalysesTypes of Analyses
• Classical Test Theory StatisticsClassical Test Theory Statistics
• IRT Model AssumptionsIRT Model Assumptions
• Model FitModel Fit
• Differential Item FunctioningDifferential Item Functioning
• Item CalibrationItem Calibration
![Page 5: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/5.jpg)
5 04/21/23
Classical Test Theory StatisticsClassical Test Theory Statistics
• Out of rangeOut of range
• Item frequencies and distributionsItem frequencies and distributions
• Inter-item correlationsInter-item correlations
• Item-scale correlationsItem-scale correlations
• Internal consistency reliabilityInternal consistency reliability
![Page 6: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/6.jpg)
6 04/21/23
IRT Model AssumptionsIRT Model Assumptions
• (Uni)dimensionality(Uni)dimensionality
• Local independenceLocal independence
• MonotonicityMonotonicity
![Page 7: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/7.jpg)
7 04/21/23
Sufficient UnidimensionalitySufficient Unidimensionality
• Confirmatory factor modelsConfirmatory factor models
One factorOne factor
Bifactor (general and group factors)Bifactor (general and group factors)
![Page 8: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/8.jpg)
8 04/21/23
Local IndependenceLocal Independence
• After controlling for dominant factor(s), item After controlling for dominant factor(s), item pairs should not be associated.pairs should not be associated.
Look at residual correlations (> 0.20)Look at residual correlations (> 0.20)
![Page 9: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/9.jpg)
9 04/21/23
MonotonicityMonotonicity
• Probability of selecting a response category Probability of selecting a response category indicative of better health should increase as indicative of better health should increase as underlying health increases.underlying health increases.
• Item response function graphs withItem response function graphs with
y-axis: proportion positive for item stepy-axis: proportion positive for item step
x-axis: raw scale score minus item scorex-axis: raw scale score minus item score
![Page 10: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/10.jpg)
10 04/21/23
![Page 11: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/11.jpg)
11 04/21/23
Category Response Curves for Category Response Curves for Samejima’s Graded Response ModelSamejima’s Graded Response Model
0.0
0.2
0.4
0.6
0.8
1.0
-3 -2 -1 0 1 2 3
Theta (q)
P(X
=k|q
)
P (X = 3|q)P (X = 1|q)
P (X = 2|q)
P (X = 4|q)
![Page 12: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/12.jpg)
12 04/21/23
Model FitModel Fit
• Compare observed and expected response Compare observed and expected response frequencies by item and response categoryfrequencies by item and response category
• Items that do not fit and less discriminating Items that do not fit and less discriminating items identified and reviewed by content items identified and reviewed by content expertsexperts
![Page 13: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/13.jpg)
13 04/21/23
Differential Item FunctioningDifferential Item Functioning
• Uniform DIF Uniform DIF
Threshold parameterThreshold parameter
• Non-uniform DIF Non-uniform DIF
Discrimination parameter Discrimination parameter
• Gender, race/ethnicity, age, diseaseGender, race/ethnicity, age, disease
![Page 14: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/14.jpg)
14 04/21/23
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-4 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5 4
Trait level
Pro
bab
ilit
y o
f "Y
es"
Res
po
nse
Location DIF Slope DIF
Dichotomous Items Showing DIF(2-Parameter Model)
White
Hispanic
Hispanic
White
![Page 15: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/15.jpg)
15 04/21/23
Item CalibrationItem Calibration
• Item parameters (threshold, discrimination)Item parameters (threshold, discrimination)
• Mean differences for studied disease groupsMean differences for studied disease groups
![Page 16: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/16.jpg)
16 04/21/23
Example of Lessons Learned in Secondary Analyses
Emotional distress
Cannot be adequately modeled as a unidimensional construct.
Limited representation of positive end of construct
Several items having some response options that provide little information.
![Page 17: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/17.jpg)
17 04/21/23
Documentation
Public website: http://www.nihpromis.org/
Peer-reviewed manuscripts, e.g.:
Hays, R. D. et al. (in press). Item response theory analyses of physical functioning items in the Medical Outcomes Study. Medical Care.
Reeve, B. B. (in press). Psychometric evaluation and calibration of health-related quality of life items banks: Plans for the Patient-Reported Outcome Measurement Information System (PROMIS). Medical Care.
![Page 18: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/18.jpg)
18 04/21/23
![Page 19: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/19.jpg)
19 04/21/23
Datasets Subjected to Psychometric AnalysisDatasets Subjected to Psychometric Analysis
Cancer Fatigue:Cancer Fatigue: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU
Cancer Pain:Cancer Pain: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU
Cancer Social:Cancer Social: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU
CSSCD:CSSCD: Cooperative Study of Sickle Cell Disease (pediatric)Cooperative Study of Sickle Cell Disease (pediatric)
CHC:CHC: Chronic Hepatitis C StudyChronic Hepatitis C Study
CHS:CHS: Cardiovascular Health StudyCardiovascular Health Study
DIG:DIG: Digitalis Investigation Group Quality of Life Sub-studyDigitalis Investigation Group Quality of Life Sub-study
IMMPACT:IMMPACT: Multiple Pain ProjectsMultiple Pain Projects
MOS:MOS: Medical Outcomes StudyMedical Outcomes Study
NGHS:NGHS: National Growth and Health Study (peds.)National Growth and Health Study (peds.)
Q-Score:Q-Score: Cancer Quality of Life Project at NWUCancer Quality of Life Project at NWU
WHOQOL:WHOQOL: World Health Organization Quality of Life ProjectWorld Health Organization Quality of Life Project
![Page 20: Psychometric Evaluation and Calibration Plan](https://reader036.vdocuments.us/reader036/viewer/2022062314/568148fe550346895db627b1/html5/thumbnails/20.jpg)
20 04/21/23
Datasets Subjected to Psychometric AnalysisDatasets Subjected to Psychometric Analysis
PROMIS Domains
Emotional Distress Fatigue Pain Physical FunctionSocial Role
Participation
CHS(NWU/Cook)
DIG(UCLA/Hays)
Q-Score(NWU/Bode)
CHC(Medtap/Revicki
& Chen)
Cancer Pain(NWU/Lai)
IMMPACT(Medtap/Revicki
& Chen)
CHS(NWU/Cook)
DIG(UCLA/Hays)
CHS(NWU/Cook)
Cancer Social(NWU/Bode)
CHS(NWU/Cook)
Cancer Fatigue(NWU/Lai)
WHOQOL(UCLA/Hays)
WHOQOL(UCLA/Hays)
MOS(UCLA/Hays& Spritzer)
Q-Score(NWU/Lai)
WHOQOL(UCLA/Hays)
WHOQOL(UCLA/Hays)
WHOQOL(UCLA/Hays)
CSSCD(NWU)
NGHS(NWU)
NGHS(NWU)