bias in clinical research: measurement bias misclassification of dichotomous exposure & outcome...
TRANSCRIPT
Bias in Clinical Research: Measurement Bias
• Misclassification of dichotomous exposure & outcome variables
– non-differential misclassification
– differential misclassification
– magnitude and direction of bias
• Misclassification of multi-level and continuous variables
– some of the rules changes
• Advanced topics
– misclassification of confounding variables
– back-calculating to the truth
Measurement Bias
• Definition– bias that is caused when the information collected
about or from subjects is not completely valid (accurate)
• any type of variable: exposure, outcome, or confounder
– aka: misclassification bias; information bias (text); identification bias
• misclassification is the immediate result of an error in measurement
Misclassification of Dichotomous Variables: Terms Related to Measurement Validity
• Sensitivity
– the ability of a measurement to identify correctly those who have the
characteristic (disease or exposure) of interest.
• Specificity
– the ability of a measurement to identify correctly those who do NOT
have the characteristic of interest
• Applies to any dichotomous variable, not just diagnoses
Gold Standard Present Absent
Your Present a b Measurement Absent c d
Sensitivity = a/(a+c) Specificity = d/(b+d)
Causes for Misclassification• Questionnaire problems
– inaccurate recall– ambiguous questions– under or overzealous interviewers
• Biological specimen collection– problems in specimen collection or processing or storage
• Biological specimen testing– inherent limits of detection
– faulty instruments • Data management problems in coding• Design or analytic problems
– incorrect time period assessed– lumping of variables (composite variables)
Diseased
Exposed
+ -
+
-
SOURCE POPULATION
STUDY SAMPLE
Non-Differential Misclassification of Exposure: Imperfect Sensitivity
Problems with sensitivity in the measurement of exposure - independent of disease status
e.g., case-control study
exposure = alcohol abuse
Evenly shaded arrows =
non-differential
Non-differential Misclassification of Exposure
Truth: No misclassification (100% sensitivity/specificity)
Exposure Cases ControlsYes 50 20No 50 80
OR= (50/50)/(20/80) = 4.0
Presence of 70% sensitivity in exposure classification
Exposure Cases ControlsYes 50-15=35 20-6=14No 50+15=65 80+6=86
OR= (35/65)/(14/86) = 3.3
Effect of non-differential misclassification of 2 exposure categories: Bias the OR toward the null value of 1.0
Diseased
Exposed
+ -
+
-
SOURCE POPULATION
STUDY SAMPLE
Non-Differential Misclassification of Exposure: Imperfect Specificity
Problems with specificity of exposure measurement - independent of disease status
e.g., exposure = self-reported second-hand smoke exposure
Non-differential Misclassification of Exposure
Truth: No misclassification (100% sensitivity/specificity)
Exposure Cases ControlsYes 50 20No 50 80
OR= (50/50)/(20/80) = 4.0
Presence of 70% specificity in exposure classification
Exposure Cases ControlsYes 50+15=65 20+24=44No 50-15=35 80-24=56
OR= (65/35)/(44/56) = 2.4
Effect of non-differential misclassification of 2 exposure categories: Bias the OR toward the null value of 1.0
Diseased
Exposed
+ -
+
-
SOURCE POPULATION
STUDY SAMPLE
Non-Differential Misclassification of Exposure: Imperfect Specificity and Sensitivity
Problems with sensitivity - independent of disease status
Problems with specificity - independent of disease status
Non-Differential Misclassification of Exposure: Imperfect Sensitivity and Specificity
Exposure Cases ControlsYes 50 20No 50 80 True OR = (50/50) / (20/80) = 4.0
True Cases Controls Distribution exp unexp exp unexp (gold standard) 50 50 20 80
Study distribution: Cases ControlsExposed 45 10 55 18 16 34Unexposed 5 40 45 2 64 66
sensitivity 0.90 0.80 0.90 0.80 or specificity
Exposure Cases ControlsYes 55 34No 45 66 Observed OR = (55/45) / (34/66) = 2.4
SOURCE POPULATION
STUDYSAMPLE
Sensitivity = 0.9
Specificity = 0.8
Non-Differential Misclassification of Exposure: Imperfect Sensitivity & Specificity and Uncommon Exposure
Exposure Cases ControlsYes 30 10No 70 190 True OR = (30/70) / (10/190) = 8.1
True Cases Controls Distribution exp unexp exp unexp (gold standard) 30 70 10 190
Study distribution: Cases ControlsExposed 27 14 41 9 38 47Unexposed 3 56 59 1 152 153
sensitivity 0.90 0.80 0.90 0.80 or specificity
Exposure Cases ControlsYes 41 47No 59 153 Observed OR = (41/59) / (47/153) = 2.3
SOURCE POPULATION
STUDYSAMPLE
e.g. radon exposure
Sensitivity = 0.9
Specificity = 0.8
Non-differential Misclassification of Exposure: Magnitude of Bias on the Odds Ratio
True OR=4.0
2.20.0770.900.90
2.80.200.900.90
3.00.3680.900.90
1.90.200.600.90
3.20.200.950.90
1.90.200.850.60
2.60.200.850.90
Observed ORPrev of Exp in controls
SpecificitySensitivity
Bias as a function of non-differential imperfect sensitivity and specificity of exposure measurement
0.9
0.7
0.5
Sensitivity of exposure measurement
Specificity of exposure measurement
Copeland et al. AJE 1977
True OR = 2.67
Prevalence of exposure in controls = 0.2
Ap
par
ent
Od
ds
Rat
io
2.8
2.5
2.2
1.9
1.6
1.3
1.0
.50 .55 .60 .65 .70 .75 .80 .85 .90 .95 1.00
Bias as a function of non-differential imperfect sensitivity and specificity of exposure measurement
0.9
0.7
0.5
Sensitivity of exposure measurement
Specificity of exposure measurement
Copeland et al. AJE 1977
True OR = 2.67
Prevalence of exposure in controls = 0.2
Ap
par
ent
Od
ds
Rat
io
2.8
2.5
2.2
1.9
1.6
1.3
1.0
.50 .55 .60 .65 .70 .75 .80 .85 .90 .95 1.00
Non-Differential Misclassification of Exposure in a Cohort Study: Effect of Sensitivity, Specificity and Prevalence of Exposure
U = sensitivity
V = specificity
Flegal et al. AJE 1986
Non-Differential Misclassification of Exposure in a Cohort Study:
Effect at Different Magnitudes of True Association
Flegal et al. AJE 1986
U = sensitivity
V = specificity
Non-Differential Misclassification of Exposure: Rules of Thumb Regarding Sensitivity & Specificity
Exposure Cases ControlsYes 50 100No 50 300 True OR = (50/50) / (100/300) = 3.0
SOURCE POPULATION
Sens + Spec = 1 gives OR = 1 (no effect)
Sensitivity Specificity ObservedOR
0.8 1.0 2.6
0.8 0.8 1.9
0.4 0.6 1.0
0.4 0.4 0.82
Sens + Spec >1 but <2 gives attenuated effect
Sens + Spec < 1 gives reversal of effect
Diseased
Exposed
+ -
+
-
SOURCE POPULATION
STUDY SAMPLE
Non-Differential Misclassification of Outcome
Problems with outcome sensitivity -independent of exposure status
Problems with outcome specificity - independent of
exposure status
Evenly shaded arrows =
non-differential
Bias as a function of non-differential imperfect sensitivity and specificity of outcome measurement in a cohort study
Sensitivity of outcome measurement0.9
0.7
0.5
Specificity of outcome measurementCopeland et al. AJE 1977
True risk ratio = 2.0
Cumulative incidence in unexposed = 0.05
Non-Differential Misclassification of Outcome: Effect of Incidence of Outcome
Copeland et al. AJE 1977
Specificity of outcome measurement
0.2 .1
0.1 0.05
0.05 0.025
Cumulative incidence of outcome
Exposed Unexposed
True risk ratio = 2.0
Sensitivity of outcome measurement held fixed = 0.9
Special Situation In a Cohort or Cross-sectional Study
Misclassification of outcome• If specificity of outcome measurement is 100%• Any degree of imperfect sensitivity, if non-differential, will not
bias the risk ratio or prevalence ratio• e.g.,
• Risk difference, however, is changed by a factor of (1 minus sensitivity), in this example, 30% (truth=0.1; biased = 0.07)
DiseaseNoDisease
Exposed 20 80 100Unexposed 10 90 100
2.0
1001010020
ratio )prevalence (or Risk
DiseaseNoDisease
Exposed 20-6=14 80+6=86100Unexposed 10-3=7 90+3=93100
2.0
1007
10014
ratio )prevalence (or Risk
Truth
70% sensitivity
When specificity of outcome is 100% in a cohort or cross sectional study
Sensitivity of outcome measurement0.9
0.7
0.5
Specificity of outcome measurementCopeland et al. AJE 1977
True risk ratio = 2.0
Cumulative incidence in unexposed = 0.05
In contrast, 100% specificity of exposure measurement still results in bias
0.9
0.7
0.5
Sensitivity of exposure measurement
Specificity of exposure measurement
Copeland et al. AJE 1977
True OR = 2.67
Prevalence of exposure in controls = 0.2
Ap
par
ent
Od
ds
Rat
io
2.8
2.5
2.2
1.9
1.6
1.3
1.0
.50 .55 .60 .65 .70 .75 .80 .85 .90 .95 1.00
When specificity of outcome measurement is 100% in a cohort or cross sectional study
• Worth knowing about when choosing cutoff for continuous variables on ROC curves
• Choosing most specific cutoff (or 100% cutoff) will lead to least biased ratio measures of effect
0.0 0.1 0.2 0.3 0.4 0.5
0.5
0.6
0.7
0.8
0.9
1.0
1.0 0.9 0.8 0.7 0.6 0.5
Sen
sit
ivit
y
1 - Specificity
00.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
OD: 0.06 Specificity: 84 % Sensitivity: 100 %
OD: 0.19 Specificity: 95 % Sensitivity: 94 %
OD: 0.49 Specificity: 100 % Sensitivity: 74 %
0.0 0.1 0.2 0.3 0.4 0.5
0.5
0.6
0.7
0.8
0.9
1.0
1.0 0.9 0.8 0.7 0.6 0.5
Sen
sit
ivit
y
1 - Specificity
00.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
OD: 0.06 Specificity: 84 % Sensitivity: 100 %
OD: 0.19 Specificity: 95 % Sensitivity: 94 %
OD: 0.49 Specificity: 100 % Sensitivity: 74 %
OD: 0.06 Sensitivity = 100%OD: 0.06 Specificity = 84%
OD: 0.19 Sensitivity = 94%OD: 0.19 Specificity = 95%
OD: 0.49 Sensitivity = 74%OD: 0.49 Specificity = 100%
1.0 0.9 0.8 0.7 0.6 0.5
Specificity
Efficacy of a pertussis vaccine
• Outcome: Cough > 5 days– No. of events: 2672– Result: No significant difference between groups
• Outcome: Cough + microbiologic pertussis confirmation– No. of events: 10– Result: rate ratio = 0.08 (92% vaccine efficacy) (95% CI = 0.01 to 0.68)
• Acellular vaccine vs. control (hepatitis A vaccine) for the prevention of pertussis in adults (Ward et al. NEJM 2005)
Group No. of subjects Person-years Pertussis vaccine 1391 2421 Control 1390 2444
Pervasiveness of Non-Differential Misclassification
• Direction of this bias is typically towards the null
• Therefore, this is called a “conservative” bias
• Goal, however, is to get the truth
• Consider how much underestimation of effects must be occurring in research
• How many “negative” studies are truly “positive”?
Differential Misclassification of ExposureWeinstock et al. AJE 1991• Nested case-control study in Nurses Health Study cohort
• Cases: women with new melanoma diagnoses
• Controls: women w/out melanoma - by incidence density sampling
• Measurements of exposure: questionnaire about self-reported
“tanning ability”; administered shortly after melanoma development
MelanomaNoMelanoma
No tan to light tan 15 77Med to dark tan 19 157
1.6
157771915
OR
• Question asked after diagnosis
• Question asked before diagnosis (NHS baseline)
MelanomaNoMelanoma
No tan to light tan 9 79Med to dark tan 25 155
0.7
15579259
OR
MelanomaNoMelanoma
No tan to light tan 15 77Med to dark tan 19 157
1.6
157771915
OR
Melanoma
Tanningability
+ -
No
Yes
SOURCE POPULATION
STUDY SAMPLE
“Tanning Ability” and Melanoma:
Differential Misclassification of Exposure
Imperfect specificity of exposure measurement - mostly in cases
Bias away from the null
Congenital Malformation
Exposed
+ -
+
-
SOURCE POPULATION
STUDY SAMPLE
Differential Misclassification of Exposure:
Exposures During Pregnancy and Congenital Malformations
Cases more likely than controls to remember a variety of exposures
Cases might be more likely than controls to falsely state a
variety of exposures
Uneven shading of arrows =
differential
Differential Misclassification of Exposure: Magnitude of Bias on the Odds Ratio
True OR=3.9
Exposure Classification
Sensitivity Specificity
Cases Controls Cases Controls OR
0.90 0.60 1.0 1.0 5.79
0.60 0.90 1.0 1.0 2.22
1.0 1.0 0.9 0.70 1.00
1.0 1.0 0.7 0.90 4.43
Prevalence of Exposure in Controls = 0.1
Misclassification of Dichotomous Exposure or Outcome: Summary of Effects
Misclassification Effect on Ratio Measure of Association
Non-differential Exposure Towards null Outcome Towards null*
Differential
Exposure Away or towards null Outcome Away or towards null
*Exception: When specificity is 100%, no effect on risk ratio or prevalence ratio regardless of sensitivity
Non-differential Misclassification of Multi-level Exposure
Cases ControlsOddsRatio
None 100 100 1.0
Low 200 100 2.0
High 600 100 6.0
E x p o s u r e
Cases ControlsOddsRatio
None 100 100 1.0
Low 440 140 3.1
High 360 60 6.0
Misclassification between adjacent exposure categoriesTruth
Bias away from the nullDosemeci et al. AJE 1990
Misclassification of Multi-level Exposure
Cases ControlsOddsRatio
None 100 100 1.0
Low 200 100 2.0
High 600 100 6.0
E x p o s u r e
Cases ControlsOddsRatio
None 420 180 1.0
Low 120 60 0.86
High 360 60 2.57
Misclassification between adjacent and non-adjacent exposure categories
Truth
Appearance of J-shaped relationshipDosemeci et al. AJE 1990
Relating the Reproducibility and Validity of Measurements to Measurement Bias --
Categorical Variables
• Validity – how sensitivity and specificity of a measurement results in measurement bias covered in prior slides
• How about reproducibility? – Recall that a measurement with imperfect
reproducibility will lack perfect validity (unless it is repeated many many times)
Reproducibility and Validity of a Measurement
With only one shot at the measurement, most of the time you will be off the center of the target
GoodB-Ball
PoorB-Ball
>6 ft 10 30 40 +1 10 +3 30<6 ft 10 50 60 10 +1 50 +5
20 80 100 20 80
P
GoodB-Ball
PoorB-Ball
>6 ft 10 32 42<6 ft 10 48 58
20 80 100
Truth = Prevalence Ratio= (10/40) / (10/60) = 1.5
Observed = Prevalence Ratio = (10/42) / (10/58) = 1.38
10% Misclassification
Imperfect reproducibility leads
to 90% sensitivity and 90% specificity of
height measurement –non-differential with respect to outcome
Relating the Reproducibility and Validity of Measurements to Measurement Bias – Interval Scale (Continuous) Variables
Validity (Systematic error):• Response moves systematically up or down the scale; no real
effect in analytic studies
Reproducibility (Random error):
Assuming:
• Exposure is normally distributed with variance, 2True
• Random error is normally distributed with variance, 2E
• Then, the observed regression coefficient is equal to the true regression coefficient times:
• i.e., the greater the measurement error, the greater the attenuation (bias) towards the
null
22
2
ETrue
True
(i.e. reproducibility)
Measured Value
Truth
Advanced Topics
• Misclassification of confounding variables– net result is failure to fully control (adjust) for that variable (left with
residual confounding)– measures of association may be over or under-estimated
• Back-calculating to unbiased results– thus far, truth about relationships have been assumed– in practice, we just have observed results– when extent of classification errors (e.g., sensitivity and specificity)
are known, it is possible to back-calculate to truth– if exact classification errors are not known, it is possible to perform
sensitivity analyses to estimate a range of study results given a range of possible classification errors
Poor Reproducibility
Poor Validity
Good Reproducibility
Good Validity
Managing Measurement Bias
• Prevention and avoidance are critical– study design phase is critical; little to be done after study over
• Become an expert in the measurement of your primary variables
• For the other variables, seek out the advice of other experts
• Optimize the reproducibility/validity of your measurements!