applying evidence-based medicine to laboratory test...
TRANSCRIPT
![Page 1: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/1.jpg)
Applying Evidence-Based Medicine
to Laboratory Test Selection
Robert Schmidt MD, PhD, MBA
![Page 2: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/2.jpg)
Motivation
• Value for money
– Improved outcomes
– Lower costs
• Changes in health care management
![Page 3: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/3.jpg)
Impact of Laboratory Testing
3
Testing
Medical Costs
Laboratory tests account for 3% of
medical costs but affect 70% of medical
decisions
![Page 4: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/4.jpg)
Webinar Topics
• Hierarchy of evidence in test evaluation
• How to evaluate the utility of tests
• Gaps in evidence
• Identification of misused tests
– examples
![Page 5: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/5.jpg)
Hierarchy of Effectiveness
5
Analytical performance
Clinical performance
Clinical effectiveness
Cost effectiveness
Societal Impact
![Page 6: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/6.jpg)
Analytical Performance
• Limit of detection
• Precision
• Linear Range
• Accuracy
• Interferences
• Cost
• Operational capability
– Reliability/maintainability/durability
– Turnaround time
![Page 7: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/7.jpg)
Hierarchy of Effectiveness
7
Analytical performance
Clinical performance
Clinical effectiveness
Cost effectiveness
Societal Impact
![Page 8: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/8.jpg)
Clinical Performance
• Diagnostic Accuracy
• Does the test discriminate those with disease
from those without?
![Page 9: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/9.jpg)
What is a Diagnostic Test
Accuracy Study?
• Population
• Index Test
• Comparator (Reference Test)
• Outcome
• Timing
• Setting
9
![Page 10: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/10.jpg)
Basic Accuracy Statistics
10
Index Test
Reference Test
Positive Negative
Positive TP FP
Negative FN TN
Sensitivity = TP /(TP + FN)
Specificity = TN /(TN + FP)
Positive Predictive Value = PPV = TP / (TP + FP)
Negative Predictive Value = NPV = TN /( TN + FN)
![Page 11: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/11.jpg)
Framework for Study Evaluation
Study
Value
Internal
Validity
External
Validity
Do the results of this
study actually
measure what they
purport to measure?
Are the results of
this study
applicable to my
clinical question?
![Page 12: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/12.jpg)
Threats to Validity
Study
Value
Internal
Validity
External
Validity
Do the results of
this study
actually measure
what they purport
to measure?
Are the results
of this study
applicable to my
clinical
question?
Bias Precision Real
Differences
![Page 13: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/13.jpg)
Sources of
Variation
Study
Value
Internal
Validity
External
Validity
Bias Precision Real
Differences
Threats to Validity
![Page 14: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/14.jpg)
14
Bias Precision External Validity
![Page 15: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/15.jpg)
External Validity
15
Population
Index Test
Reference Test
Outcome
Population
Index Test
Reference Test
Outcome
Clinical Problem Potentially Relevant
Study
![Page 16: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/16.jpg)
Variability of Study Results 65 DTA studies of parotid gland FNA
Schmidt RL, Hall BJ, Wilson AR, Layfield LJ. A Systematic Review and Meta-analysis of the Diagnostic Accuracy of FNAC for Parotid Gland Lesions. Am J Clin Pathol. 2011;136(1):45-59.
Bias?
Precision?
Real Differences?
![Page 17: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/17.jpg)
17
120 heme
patients
Measure BDG
EORTC
criteria
Study A
100 culture
positive pts
(cases)
Measure BDG
100 medical
students (controls)
Study B
Patient Factors (Beta-D glucan for Diagnosis of Invasive Fungal Disease)
![Page 18: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/18.jpg)
negative positive
Study A
negative positive
Study B
120 heme onc
patients
Measure BDG
EORTC
criteria
Study A
100 culture positive
pts
(cases)
Measure BDG
100 medical students
(controls)
Study B
Patient Spectrum
![Page 19: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/19.jpg)
What factors affect patient
spectrum?
19
![Page 20: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/20.jpg)
Impact of referral patterns on patient spectrum
Primary Care
negative positive
negative positive
Specialist
“Easy” Diagnoses
![Page 21: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/21.jpg)
Index Test
• Applicability
– Definition of the test
• Sources of Bias
– Test Review Bias
– Reading Order Bias
– Incorporation Bias
21
![Page 22: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/22.jpg)
Index test Definition
• What is the test?
– Isolated index test?
– Isolated index test plus clinical information?
– Isolated test plus previous testing?
22
![Page 23: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/23.jpg)
Test Definition: Impact of Additional Information
23
Patients
Index Test
Reference
Test
Positive Negative
Other Tests
Reference
Test
![Page 24: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/24.jpg)
Reference Test
Misclassification Bias
• Error in the reference standard
– (Brass Standard vs Gold Standard)
– Nondifferential misclassification
• Error rate independent of index test result
• Almost always reduces sensitivity and specificity
– Differential misclassification
• Error rate depends on index test result
– (e.g. errors higher for cases with a positive result)
• Impact on accuracy statistics difficult to predict
24
![Page 25: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/25.jpg)
Example: Nondifferential Misclassification
10% Misclassification Rate
Histopathology
Positive Negative Total
FNA Positive 820* 180 1000
Negative 180 820 1000
Sensitivity = 90%
Specificity = 90%
Sensitivity = 82%
Specificity = 82%
Histopathology
Positive Negative Total
FNA Positive 900 100 1000
Negative 100 900 1000
Sn = 90%
Sp = 90%
Sn = 82%
Sp = 82%
90
10
*Example: 820 = 900 (1-0.1) + 0.1 (100)
![Page 26: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/26.jpg)
Verification Bias
26
Patients
Index
Test
Reference Test
Reference Test
Nothing
Positive Negative
Nothing
α 1-α β 1-β
α = positive verification rate
β = negative verification rate
![Page 27: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/27.jpg)
Study Population
Positive Negative
FNA Positive 180 80
Negative 20 720
Actual Sensitivity = 90%
Actual Specificity = 90% Observed Sensitivity = 99%
Actual Specificity = 50%
Example of Verification Bias:
1000 people with a lump
Prevalence of neoplasia = 20%
FNA Sensitivity = 90%
FNA Specificity = 90%
90% of positive cases get histologic follow-up
10% of negative cases get histologic follow-up
Actual Sensitivity = 90%
Actual Specificity = 90%
Observed Sensitivity = 99%
Observed Specificity = 50%
Verified Population
Positive Negative
FNA Positive 162 72
Negative 2 72
![Page 28: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/28.jpg)
Impact of partial verification on bias
R = β/α
![Page 29: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/29.jpg)
Differential Verification Bias (Work-up bias)
29
Patients
Index Test
Reference Test 1
Reference Test 2
Positive Negative
Clinical Follow-up Histopathology
![Page 30: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/30.jpg)
Observed Results Reference Test
Positive Negative
FNA Positive 179 81
Negative 90 650
True Results Reference Test
Positive Negative
FNA Positive 180 80
Negative 20 720
Actual Sensitivity = 90%
Actual Specificity = 90% Observed Sensitivity = 99%
Actual Specificity = 50%
Example: Differential verification bias
1000 people with a lump
Prevalence of neoplasia = 20%
FNA Sensitivity = 90%
FNA Specificity = 90%
positive cases get histologic follow-up (error rate: 0.01%)
positive cases get clinical follow-up (error rate: 10%)
Actual Sensitivity = 90%
Actual Specificity = 90%
Observed Sensitivity = 66%
Observed Specificity = 89%
Usually causes negative bias in sensitivity
![Page 31: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/31.jpg)
Indeterminate Results
• How should they be included?
• How do they affect accuracy statistics?
31
Study A
Index Test
Reference Test
Total POS IND NEG
POS 80 1 19 100
IND 3 5 2 10
NEG 13 2 85 100
Total 96 8 106 210
Study B
Index Test
Reference Test
Total POS IND NEG
POS 80 0 2 82
IND 15 8 19 42
NEG 1 0 85 86
Total 96 8 106 210
![Page 32: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/32.jpg)
Impact of Indeterminate Rate on Accuracy
Scenario A
Index Test
Reference Test
Total POS IND NEG
POS 80 1 19 100
IND 3 5 2 10
NEG 13 2 85 100
Total 96 8 106 210
Scenario B
Index Test
Reference Test
Total POS IND NEG
POS 80 0 2 82
IND 15 8 19 42
NEG 1 0 85 86
Total 96 8 106 210
negative positive
negative positive
![Page 33: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/33.jpg)
Other Problems With Accuracy
Studies
• Inaccurate Reporting
• Tests are viewed independently
– Key question: how does test information impact
likelihood of disease?
• Heterogeneity
![Page 34: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/34.jpg)
Variability of Study Results 65 DTA studies of parotid gland FNA
Schmidt RL, Hall BJ, Wilson AR, Layfield LJ. A Systematic Review and Meta-analysis of the Diagnostic Accuracy of FNAC for Parotid Gland Lesions. Am J Clin Pathol. 2011;136(1):45-59.
Bias?
Precision?
Real Differences?
![Page 35: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/35.jpg)
Hierarchy of Evidence for Clinical Studies
Cochrane Reviews
![Page 36: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/36.jpg)
The State of Nature
As revealed by experiments:
As revealed by publications:
As selected for review:
As synthesized
Quality Search Strategy Sensitive Multiple data bases Unpublished sources
Well Defined Criteria Inclusion Exclusion
Defined a priori Reproducible
Systematic Review:
Meta-Analysis
Meta-Analysis of DTA studies
![Page 37: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/37.jpg)
The State of Nature
As revealed by experiments:
θ sensitivity
1XnX
nX
As revealed by publications:
As selected for review:
As synthesized
θ sensitivity
![Page 38: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/38.jpg)
The State of Nature
As revealed by experiments:
θ sensitivity
1XnX
nX
As revealed by publications:
As selected for review:
As synthesized
θ sensitivity
![Page 39: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/39.jpg)
![Page 40: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/40.jpg)
Comparison of outcomes of published vs registered
RCTs
Simes J Stat Med; 1987
![Page 41: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/41.jpg)
Effect of various outcomes on publication rate
![Page 42: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/42.jpg)
Bottom line
• Lots of deficiencies in the literature
– Incomplete reporting (STARD)
– Biased results
– Inaccurate reporting of results
• Problems with meta-analysis
– Publication bias (for clinical studies)
– Meta-analysis of non-comparative studies
– Heterogeneity
– GIGO
– Relatively few available
• Accuracy is a Surrogate Measure: Not Linked to Value
![Page 43: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/43.jpg)
Hierarchy of Effectiveness
43
Analytical performance
Clinical performance (diagnostic accuracy)
Clinical effectiveness
Cost effectiveness
Societal Impact
![Page 44: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/44.jpg)
Clinical Utility
• Degree to which a test is associated with
improved outcomes
• Do tests change outcomes that matter to
patients?
![Page 45: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/45.jpg)
Components of Utility
• Medical Impact
– Change in management:
• stop, start, modify or withdraw treatment
– Effect of test on patients (adverse events)
• Emotional Impact
• Social Impact
A test can have clinical utility without medical impact
![Page 46: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/46.jpg)
Characteristics of Clinical Utility
• Probabilistic
– Outcome is not assured even if test is perfect
• Relative
– No absolute scale
– Defined relative to an alternative
• Contextual
– Utility depends on:
• Available treatments
• Alternative tests
• Constantly changing
![Page 47: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/47.jpg)
Measurement of Utility
• Utility Scale
– Dead = 0
– Best possible health = 1
– Intermediate health states: between 0 and 1
• Quality adjusted life years (QALY)
– Accounts for:
• Quantity of life (years)
• Quality of life (Utility)
![Page 48: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/48.jpg)
Measuring Utility
?
Perfect Health
(Utility=1)
Dead
Utility = 0
p
1-p
COPD (Utility=?)
Expected Utility of COPD = p*1 + (1-p)*0 = p = 0.7?
![Page 49: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/49.jpg)
Measuring Utility
?
Perfect Health
(Utility=1)
Dead
Utility = 0
p
1-p
Common Cold
(Utility=?)
Expected Utility of COPD = p*1 + (1-p)*0 = p = 0.999?
![Page 50: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/50.jpg)
Add up QALYs over life
• Life after test A: expected QALYs = 9.5
• Life after test B: expected QALYs = 8.0
• Test A provides a benefit of 1.5 QALYs relative to
Test B
![Page 51: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/51.jpg)
Where does data come from?
• Randomized Clinical Trials
– Best Evidence for Utility
• Modeling
![Page 52: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/52.jpg)
Diagnostic Randomized Controlled Trial
Patients R
Test A
Test B
Treatment
No Treatment
Treatment
No Treatment
Outcomes
Outcomes
Outcomes
Outcomes
positive
negative
positive
negative
![Page 53: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/53.jpg)
Problems with DRCTs
• Costly
• Time consuming
• Inefficient
• Indirect
![Page 54: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/54.jpg)
Evidence from DRCTs
• Very few published studies
– 37 DRCTs per year
– 11,000 RCTs
![Page 55: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/55.jpg)
Do we need DRCTs?
• Not always
• Combine evidence from test performance with
evidence from therapeutic trials
Accuracy
Study RCT
Utility
Calculation
Treatment
Outcomes
Net Reclassification
![Page 56: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/56.jpg)
Do we always need DRCTs?
• Test A has utility relative to Test B if:
– Better sensitivity but same specificity
– Better specificity but same sensitivity
– Same sensitivity and specificity but fewer adverse
events
![Page 57: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/57.jpg)
Modeling
Good
Health Disease Death
Information required:
• List of states
• Utility of each state
• Transition probabilities
Output:
• QALYs
![Page 58: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/58.jpg)
Good
Health Disease Death
Good
Health Disease Death
Good
Health Disease Death
Good
Health Disease Death
Test
Treat
Positive
Don’t
Treat
Negative
TP
FP
FN
TN
Expected
QALYs
![Page 59: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/59.jpg)
Hierarchy of Effectiveness
59
Analytical performance
Clinical performance
Clinical effectiveness
Cost effectiveness
Societal Impact
![Page 60: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/60.jpg)
Cost Effectiveness Modeling
Alternative QALYs Total Cost
Test A 11.0 $25,000
Test B 9.0 $15,000
Difference 2.0 $10,000
Incremental Cost Effectiveness Ratio (ICER) = Value for Money
𝐶𝑂𝑆𝑇𝐴−𝐶𝑂𝑆𝑇𝐵
𝑄𝐴𝐿𝑌𝐴 −𝑄𝐴𝐿𝑌𝐵=
Δ𝐶𝑜𝑠𝑡
Δ𝑈𝑡𝑖𝑙𝑖𝑡𝑦=
$10,000
2.0 𝑄𝐴𝐿𝑌=
$5000
𝑄𝐴𝐿𝑌
![Page 61: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/61.jpg)
Cost Effectiveness Plane
Δ𝑄𝐴𝐿𝑌 Δ
𝐶𝑂
𝑆𝑇
![Page 62: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/62.jpg)
Δ𝑄𝐴𝐿𝑌
Δ𝐶𝑂𝑆𝑇
Winners
Losers: ?????
?????
![Page 63: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/63.jpg)
Δ𝑄𝐴𝐿𝑌
Δ𝐶𝑂𝑆𝑇
Winners
Losers:
A $5000/QALY
![Page 64: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/64.jpg)
Δ𝑄𝐴𝐿𝑌
Δ𝐶𝑂𝑆𝑇
Winners
Losers
A
WTP = $50,000
ICER = $5,000
![Page 65: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/65.jpg)
Δ𝑄𝐴𝐿𝑌
Δ𝐶𝑂𝑆𝑇
A
B
WTP = $50,000
![Page 66: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/66.jpg)
Δ𝑄𝐴𝐿𝑌
Δ𝐶𝑂𝑆𝑇
A
B
WTP = $50,000
C
E
F
Z
G
H
Which new tests do you choose?
I
J
K
Y
![Page 67: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/67.jpg)
Δ𝑄𝐴𝐿𝑌
Δ𝐶𝑂𝑆𝑇
A
B
WTP = $50,000
C
E
F
Z
G
H
I
J
K
Y
C
A
D
B
Test
Other Project
![Page 68: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/68.jpg)
Modeling
• Faster, Less expensive that DRCTs
• Won’t find the unexpected
• Requires many assumptions
• Low output of studies
– 147 of 2000 cost effectiveness studies were on
diagnostic testing
![Page 69: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/69.jpg)
Hierarchy of Effectiveness
69
Analytical performance
Clinical performance
Clinical effectiveness
Cost effectiveness
Societal Impact
![Page 70: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/70.jpg)
Many steps to evaluate tests….
70
Analytical performance
Clinical performance
Clinical effectiveness
Cost effectiveness
Societal Impact
![Page 71: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/71.jpg)
• Advanced Cardiac Markers
Limited Incremental Cost
Effectiveness
![Page 72: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/72.jpg)
Ordering Errors
• 1, 25 dihydroxy vitamin D vs 25 hydroxy vitamin D
![Page 73: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/73.jpg)
New Tests with better
performance
• Celiac Disease
– ttG vs endomysial antibody
• Helicobacter pylori infection
– Stool antigen vs serology
• Pheochromocytoma
– Metanephrines vs catecholamines
![Page 74: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/74.jpg)
Tests with limited clinical use
• rT3
• Vitamin D2 and D3
• Phosphatidylserine antibodies for APS
• MBP for multiple sclerosis
![Page 75: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/75.jpg)
Deviations from guidelines
• Free PSA when total PSA > 10 or PSA < 2.5
ng/mL
• PSA screening in men over 75
![Page 76: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/76.jpg)
• Over testing
– IgA and IgG ttG for celiac disease
• Wrong context
– IgG subclasses by non-specialists
• Odd patterns
– Hospital X accounts for 3% of our volume but 70% of
the orders for Test Y
![Page 77: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/77.jpg)
Utilization Iceberg
Defensive Medicine
Ineffective ordering
Tests without followup
??????
??????
??????
Easy cases
Failure to test
![Page 78: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/78.jpg)
Conclusion
• Many paths to low utility
• Evidence base is poor
– Poor link between testing and outcomes
– Few clinical trials or modeling studies
– Problems with accuracy studies
• New tests are developed faster than they can be
evaluated
– “omics” tsunami
• Findings are transient
![Page 79: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/79.jpg)
Future
• Evidence base is accelerating
• Diagnostic accuracy literature is improving
• Need for efficient use of comparative
effectiveness studies
![Page 80: Applying Evidence-Based Medicine to Laboratory Test Selectionarup.utah.edu/media/evidence_dx/Evidence-Based Medicine.pdf · 2014-04-10 · 90% of positive cases get histologic follow-up](https://reader034.vdocuments.us/reader034/viewer/2022050110/5f4828462b2b2a02e826dfe7/html5/thumbnails/80.jpg)
P.A.C.E.®/FL Password:
ED92613
Go to www.aruplab.com/evidence-dx
and click on the
P.A.C.E.®/FL Credit Redemption Link Credit redemption for this webinar will be available through October 10, 2013
This webinar can be viewed after November 1, 2013 at ww.arup.utah.edu
where CME/SAM, P.A.C.E.® and Florida continuing education credit will
be available.