measurement: reliability and validity measures
TRANSCRIPT
Copyright 2007, The Johns Hopkins University and Jonathan Weiner. All rights reserved. Use of these materials permitted only in accordance with license rights granted. Materials provided “AS IS”; no representations or warranties provided. User assumes all responsibility for use, and all liability related thereto, and must independently review all materials for accuracy and efficacy. May contain materials owned by others. User is responsible for obtaining permissions for use from third parties as needed.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License. Your use of this material constitutes acceptance of that license and the conditions of use of materials on this site.
Measurement: Reliability and Validity Measures
Jonathan Weiner, DrPHJohns Hopkins University
Section A
Definitions and Reliability
4
Measurement
Measurement is a systematic, replicable process by which objects or events are quantified and/or classified with respect to a particular dimensionThis is usually achieved by the assignment of numerical values
5
Levels of Measurement
1.
Nominal measure
2.
Ordinal measure
3.
Interval measure
4.
Ratio measure
6
Measurement Reliability
Reliability of a MeasureThe degree to which a measurement technique can be depended upon to secure consistent results upon repeated application−
“The rubber ruler issue”
7
Measurement Validity
Validity of a MeasureThe degree to which any measurement approach or instrument succeeds in describing or quantifying what it is designed to measure−
“The 35-inch yardstick issue”
8
So, Variation in a Repeated Measure
Can be due to the following reasons:
1.
Chance or unsystematic events
2.
Systematic inconsistency
3.
Actual change in the underlying event being measured
9
Measurement Reliability
Not just the property of an instrumentRather, a measure or instrument has a certain degree of reliability when applied to certain populations under certain conditions
10Source: Suchman
Sources of “Unsystematic”
Threats to Reliability
1.
Subject reliability—factors due to research subject (e.g., patient fatigue, mood)
2.
Observer reliability—factors due to observer/
rater/interviewer (e.g., abilities of interviewer, different opinions)
3.
Situational reliability—conditions under which measurements are made (e.g., busy day at the clinic, new management)
11
“Unsystematic”
Threats to Reliability
4.
Instrument reliability—the research instrument or measurement approach itself (e.g., poorly worded questions, quirk in mechanical device)
5.
Data processing reliability—manner in which data are handled (e.g., miscoding)
12
How Do We Evaluate Observer Measurement Reliability?
Inter-rater agreement−
Compare two or more of the observers/raters at a point in time
−
Percentage of overall agreement, KappaTest-retest−
Compare measurements made by the same observer/rater at two points in time
−
Timeframe should be short enough that the construct itself hasn’t changed
13
Calculating Kappa
Rater 1Yes
Rater 1No
Rater 2Yes
A B N3
= A+B
Rater 2No
C D N4
= C+D
N1
= A+C N2
= B+D N
Kappa = Observed Agreement –
Expected Agreement due to Chancekappa = (Oa –
Ea ) / (N -
Ea), where Ea = sum of expected counts in cells A and D:
[(N1
* N3
)/N] + [(N2
* N4
)/N] (rounded to nearest whole number)Oa = sum of observed count in cells A and D
and N is the total number of respondent pairs
14
Level of Agreement Kappa Value
Almost PerfectSubstantialModerateFairSlightPoor
0.81-1.000.61-0.800.41-0.600.21-0.400.00-0.20
<0.00
Source: Landis, J.R., Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics. 33:159.
Criteria for Interpreting Kappa Statistics
Section B
Measurement: Reliablilty and Validity of Measures
16
How Do We Evaluate Instrument Reliability?
General "congruence" of instrument/ questionnaire (at same point in time)−
Item-total correlation
−
Internal consistency—the extent to which the items in a scale “hang together”
(Cronbach’s
coefficient or “alpha”
statistic)
17
How Do We Evaluate Instrument Reliability?
General "congruence" of instrument/ questionnaire (at same point in time)−
Item-total correlation
−
Internal consistency—the extent to which the items in a scale “hang together”
(Cronbach’s
coefficient or “alpha”
statistic)Approaches developers use to assess / improve congruence and efficiency−
Split half (split questionnaire into two parts)
−
Short form-long form (two different lengths)Test-Retest−
Similar to observer re-test
18
Calculating Cronbach’s
Alpha
Reliability Measure for Multi-Item ScalesTotal scale variance = sum of item variances and all item covariances[k/(k-1)]* [1- (sum of item variances/total scale variance)−
Where k = number of items
Range between 0 and 1Criteria for assessment−
≥
0.70 = adequate reliability for group comparisons
−
≥
0.90 = adequate reliability for individual monitoring
19
Relationship between Reliability and Validity
They are closely inter-dependentThere can not be validity without reliabilityThere can be reliability without validity
20
Measurement Validity (Recap)
Validity of a measure−
The degree to which any measurement approach or instrument succeeds in describing or quantifying what it is designed to measure
−
Validity reflects those errors in measurement that are systematic or constant
21
The Term “Validity”
Has More than One Connotation
The general concept of “validity” is broader than just “validity of approaches to measurement”In general, measurement reliability and validity issues fall into Campbell and Stanley’s “instrumentation” category
22
How to Evaluate Measurement Validity
Face Validity−
Measurement is accepted by those concerned as being logical on the "face of it" (also expert validity)
Content Validity−
Do the items included in the measure adequately represent the universe of questions that could have been asked?
23
How to Evaluate Measurement Validity
Criterion-Related ValidityDoes the new measure agree with an external criterion, e.g., an accepted measure?Predictive evidence−
Predictive of future event or outcome of interest
Concurrent evidence−
Correlation with “gold standard”
at the same point in
time−
Shortened scale with full scale
24
How to Evaluate Measurement Validity
Construct validity—is the measure consistent with the theoretical concept being measured?−
All tests of validity ultimately designed to support/refute the instrument’s construct validity
−
Construct validity never fully established
25
Assessing Construct Validity
Convergent evidence−
Demonstrate that your measure correlates highly (.5-.7) with measures of the same construct
−
Groups known to differ along construct have significantly different scores on measure
Discriminant evidence−
Low correlation with instruments measuring a different construct; or differences between known groups
Factorial evidence−
Clustering of items supports the theory-based grouping of items
26
Some Advanced Measurement Terms/Topics
Field of “psychometrics” is advancingWell developed “scales” and “composite indices” are available for most measures−
For example, Short-Form (SF) 36, Euro-QoL
−
Many advantages to well standardized robust multi-item measures
Supported by computer generated questions in the field of “Item Response Theory”IRT is gaining popularity (also known as “adaptive” testing)−
If a patient can walk a mile, why ask if they can walk 100 yards?
27
Practical Application of Measurement Reliability and Validity—Factors to Consider
How much time and money do you have to carry out your own tests?How small a difference in the measurement do you expect?Can you use a previously validated measure?Does the previous measure work within the context of your setting?