![Page 1: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/1.jpg)
Learning Objectives
1. Learn how to calculate basic accuracy statistics such as sensitivity, specificity, likelihood ratios and AUC
2. Understand reasons for differences in diagnostic accuracy: real differences, bias, random variation, cut-offs.
3. Understand the difference between tests conducted under ideal conditions vs real conditions
4. Understand the role of higher-level approaches to performance evaluation
![Page 2: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/2.jpg)
No Disclosures
![Page 3: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/3.jpg)
Testing a Test : Beyond Sensitivity and Specificity:
Robert Schmidt MBA MD PhD MMed
![Page 4: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/4.jpg)
Tests are Central to Medicine
• Diagnosis
• Prognosis
• Monitoring
• Management
![Page 5: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/5.jpg)
Tests Exert Great Leverage
Medical Costs Lab
Tests
3%
![Page 6: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/6.jpg)
Hierarchy of Effectiveness
6
Analytical performance
Clinical performance
Clinical effectiveness
Cost effectiveness
Societal Impact
![Page 7: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/7.jpg)
What this talk is about
Evaluating Tests: • Accuracy • Usefulness • Test Comparisons • Limitations • Future Directions
Analytical performance
Clinical performance
Clinical effectiveness
Cost effectiveness
Societal Impact
![Page 8: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/8.jpg)
Case:
Your father has just returned from his annual physical. His doctor suggested that he consider a prostatic specific antigen (PSA) test to screen for prostate cancer. He is unsure what to do and asks your advice. Should he take the test?
![Page 9: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/9.jpg)
Test result
5 10 15 20 25 30 35
The basic task: classification
Disease present No Disease
Test positive
Test negative
Test cutoff = 18
![Page 10: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/10.jpg)
Test value 5 10 15 20 25 30 35
Disease present No Disease
Disease
Test Present Absent Total
Positive 200 0 200
Negative 0 200 200
Total 200 200
Perfect Test:
![Page 11: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/11.jpg)
Test value
5 10 15 20 25 30 35
Disease present
No Disease
Positive Negative
Disease
Test Present Absent Total
Positive 148 40 188
Negative 52 160 212
Total 200 200
Not so Perfect Test:
![Page 12: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/12.jpg)
Test value
5 10 15 20 25 30 35
Disease present
No Disease
Positive Negative
True Positives
Disease
Test Present Absent Total
Positive 148 40 188
Negative 52 160 212
Total 200 200
![Page 13: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/13.jpg)
Test value
5 10 15 20 25 30 35
Disease present
No Disease
Positive Negative
False negatives
Disease
Test Present Absent Total
Positive 148 40 188
Negative 52 160 212
Total 200 200
![Page 14: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/14.jpg)
Test value
5 10 15 20 25 30 35
Disease present
No Disease
Positive Negative
True negatives
Disease
Test Present Absent Total
Positive 148 40 188
Negative 52 160 212
Total 200 200
![Page 15: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/15.jpg)
Test value
5 10 15 20 25 30 35
Disease present
No Disease
Positive Negative
False positives
Disease
Test Present Absent Total
Positive 148 40 188
Negative 52 160 212
Total 200 200
![Page 16: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/16.jpg)
Disease present
No Disease
Test value
5 10 15 20 25 30 35
Positive Negative
Disease
Test Present Absent Total
Positive 148 40 188
Negative 52 160 212
Total 200 200
How well did we classify those with disease?
Sensitivity = 148/200 = 74%
![Page 17: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/17.jpg)
Disease present
No
Disease
Test value
5 10 15 20 25 30 35
Positive Negative
Disease
Test Present Absent Total
Positive 148 40 188
Negative 52 160 212
Total 200 200
How well did we classify those without disease?
Specificity = 160/200 = 80%
![Page 18: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/18.jpg)
Disease
Test Present Absent Total
Positive 148 40 188
Negative 52 160 212
Total 200 200
5 10 15 20 25 30 35
Disease present
No Disease
Sensitivity = accuracy in the diseased group Specificity = accuracy in the nondiseased group
![Page 19: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/19.jpg)
Two useful mnemonics*
SnNout:
SpPin:
High Sensitivity Test with a Negative result
rules out
High Specificity Test with a positive result
rules in
*Guyatt & Sackett, Guide to EBM, 2002
![Page 20: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/20.jpg)
PSA test
• Sensitivity 90%
• Specificity 20%
How might this test be useful? • SnNout? • SpPin?
![Page 21: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/21.jpg)
Sensitivity of PSA Tests
Results May Vary……
![Page 22: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/22.jpg)
Specificity of PSA Test
![Page 23: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/23.jpg)
Sensitivity and Specificity Depend on Cutoff Values
Cutoff A Cutoff B
Sensitivity 74% 22%
Specificity 80% 98%
![Page 24: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/24.jpg)
5 10 15 20 25 30 35
Disease present
No Disease
Disease
Test Present Absent Total
Positive 148 40 188
Negative 52 160 212
Total 200 200
Sensitivity = 74% Specificity = 80%
![Page 25: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/25.jpg)
5 10 15 20 25 30 35
Disease present
No Disease
Threshold Effects on Test Performance
Sensitivity = 22% Specificity = 98%
Disease
Test Present Absent Total
Positive 44 4 88
Negative 156 196 312
Total 200 200
![Page 26: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/26.jpg)
5 10 15 20 25 30 35
Disease present
No Disease
Sensitivity = 98% Specificity = 22%
Disease
Test Present Absent Total
Positive 196 156 312
Negative 4 44 88
Total 200 200
Threshold Effects on Test Performance
![Page 27: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/27.jpg)
Tradeoff: Specificity vs Sensitivity
Threshold Sensitivity Specificity
15 98 22
18 74 80
20 22 98
![Page 28: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/28.jpg)
How to Compare Tests Sensitivity
Specificity Test A Test B
0.1 0.98 0.95
0.2 0.97 0.91
0.3 0.95 0.88
0.4 0.91 0.78
0.5 0.87 0.70
0.6 0.80 0.56
0.7 0.70 0.40
0.8 0.55 0.20
0.9 0.38 0.10
1.0 0.10 0.01
![Page 29: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/29.jpg)
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
Test B
Test A
Receiver Operating Characteristic (ROC) Curve
Sensitivity
Specificity Test A Test B
0.1 0.98 0.95
0.2 0.97 0.91
0.3 0.95 0.88
0.5 0.87 0.70
0.6 0.80 0.56
0.7 0.70 0.40
0.8 0.55 0.20
1.0 0.10 0.01
![Page 30: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/30.jpg)
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
ROC Curve for The Perfect Test
![Page 31: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/31.jpg)
1 – specificity = FPR
0.5 1.0
Sen
siti
vity
= T
PR
1.0
0.5
0.0
Perfect Test
*Heads = Positive Tails = Negative
![Page 32: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/32.jpg)
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
Perfect Test
Test Performance is Related to Area Under the Curve (AUC)
![Page 33: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/33.jpg)
Area Under the Curve (AUC)
Perfect Test AUC = 1.0
Real Test Useless Test AUC = 0.5
1- Specificity
Sen
siti
vity
0.0
1.0
1.0
![Page 34: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/34.jpg)
Does the AUC mean anything?
![Page 35: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/35.jpg)
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
Perfect Test
Does the AUC Mean Anything?
AUC = Average Sensitivity
![Page 36: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/36.jpg)
Test value, T
5 10 15 20 25 30 35
Disease No Disease
Meaning of AUC
Prob (TDisease > TNo Disease)
AUC = 1.0 perfect separation
![Page 37: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/37.jpg)
5 10 15 20 25 30 35
Disease present
No Disease
Meaning of AUC
AUC = 0.50 No separation
Prob (TDisease > TNo Disease)
![Page 38: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/38.jpg)
PSA vs PSA velocity
Area Under the Curve (AUC)
Study PSA velocity PSA Eggener, 2005 0.91 0.88 Ciatto, 2004 0.74 0.67 Berger, 2007 0.87 0.65
![Page 39: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/39.jpg)
Take home message:
Sensitivity Specificity
ROC Curves AUC
Threshold Effects No Threshold Effects
![Page 40: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/40.jpg)
How do I know if a test is useful?
![Page 41: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/41.jpg)
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
Test A
Is this test useful?
AUC = 0.8
![Page 42: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/42.jpg)
Usefulness is defined by the customer
Will this test tell me whether my
patient has Prostate Cancer?
![Page 43: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/43.jpg)
Usefulness is defined by the customer
Huh? The AUC is 0.75!
![Page 44: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/44.jpg)
The customer’s problem:
Uncertainty Certainty TEST
![Page 45: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/45.jpg)
Uncertainty Certainty TEST
Usefulness = Capabilities - Requirements
Requirements
Capabilities
![Page 46: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/46.jpg)
How to think about certainty
Odds = 𝑷𝒓𝒐𝒃(𝑫𝒊𝒔𝒆𝒂𝒔𝒆)
𝑷𝒓𝒐𝒃(𝑵𝒐 𝑫𝒊𝒔𝒆𝒂𝒔𝒆)=
𝑷
𝟏−𝑷
1:10 The Odds Scale
1:1
10:1
Low
Equal
High
![Page 47: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/47.jpg)
Capability = Change in Certainty 1:10
1:10
1:1
10:1 10:1
Odds of Disease Before test
Odds of disease After test
𝑷𝒓𝒐𝒃(𝑫𝒊𝒔𝒆𝒂𝒔𝒆)
𝑷𝒓𝒐𝒃(𝑵𝒐 𝑫𝒊𝒔𝒆𝒂𝒔𝒆)
Change
![Page 48: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/48.jpg)
LR+= 𝑷𝒓𝒐𝒃 𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆 𝒅𝒊𝒔𝒆𝒂𝒔𝒆)
𝑷𝒓𝒐𝒃(𝑷𝒐𝒔𝒊𝒕𝒊𝒗𝒆|𝒏𝒐 𝒅𝒊𝒔𝒆𝒂𝒔𝒆)=
𝟒𝟎𝟐𝟎𝟎
𝟒𝟐𝟎𝟎
= 𝟏𝟎. 𝟎
5 10 15 20 25 30 35
Disease present
No Disease
Positive Likelihood ratio, LR+
Bigger LR+ is better
What is the impact of a positive result?
![Page 49: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/49.jpg)
LR-= 𝑷𝒓𝒐𝒃 𝑵𝒆𝒈𝒂𝒕𝒊𝒗𝒆 𝒅𝒊𝒔𝒆𝒂𝒔𝒆)
𝑷𝒓𝒐𝒃(𝑵𝒆𝒈𝒂𝒕𝒊𝒗𝒆|𝒏𝒐 𝒅𝒊𝒔𝒆𝒂𝒔𝒆)=
𝟏𝟔𝟎𝟐𝟎𝟎
𝟏𝟗𝟔𝟐𝟎𝟎
= 𝟎. 𝟖
5 10 15 20 25 30 35
Disease present
No Disease
Negative Likelihood ratio, LR-
Smaller LR- is Better
What is the impact of a negative result?
![Page 50: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/50.jpg)
Key Relationship
Posterior Odds
= Prior Odds
LR x
![Page 51: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/51.jpg)
Likelihood = Impact Factor
1:10
1.1
10:1
1:10
1:1
10:1
Prior Odds Posterior Odds
𝑷𝒓𝒊𝒐𝒓 𝒐𝒅𝒅𝒔 𝑥 𝑳𝑹 = 𝑷𝒐𝒔𝒕𝒆𝒓𝒊𝒐𝒓 𝒐𝒅𝒅𝒔
𝑃𝑟𝑜𝑏(𝐷𝑖𝑠𝑒𝑎𝑠𝑒)
𝑃𝑟𝑜𝑏(𝑁𝑜 𝐷𝑖𝑠𝑒𝑎𝑠𝑒)
![Page 52: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/52.jpg)
Two ways to be certain:
𝑷𝒓𝒊𝒐𝒓 𝒐𝒅𝒅𝒔 𝒙 𝑳𝒊𝒌𝒆𝒍𝒊𝒉𝒐𝒐𝒅 𝑹𝒂𝒕𝒊𝒐 = 𝑷𝒐𝒔𝒕𝒆𝒓𝒊𝒐𝒓 𝒐𝒅𝒅𝒔
1:10
1:1
10:1
1:10
1:1
10:1
Prior Odds Posterior Odds
Exclude
Confirm
𝑷𝒓𝒐𝒃(𝑫
𝒊𝒔𝒆𝒂𝒔𝒆)
𝑷𝒓𝒐𝒃(𝑵
𝒐 𝑫𝒊𝒔𝒆𝒂𝒔𝒆)
![Page 53: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/53.jpg)
𝐿𝑅+ >𝑃𝑜𝑠𝑡𝑒𝑟𝑖𝑜𝑟𝑂𝑑𝑑𝑠
𝑃𝑟𝑖𝑜𝑟𝑂𝑑𝑑𝑠
Or:
𝐿𝑅− <𝑃𝑜𝑠𝑡𝑒𝑟𝑖𝑜𝑟𝑂𝑑𝑑𝑠
𝑃𝑟𝑖𝑜𝑟𝑂𝑑𝑑𝑠
Confirm
Exclude
A test can solve the problem if:
LRActual > LRRequired
![Page 54: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/54.jpg)
Futile test: Required LR+ = 𝑃𝑜𝑠𝑡𝑂𝑑𝑑𝑠
𝑃𝑟𝑖𝑜𝑟𝑂𝑑𝑑𝑠=
10
2= 5
I have a slight hunch that this person may have Prostate CA. I will do a biopsy if the odds in favor of disease are around 10:1
1:10
1:1
10:1
1:10
1:1
10:1
Prior Odds Posterior Odds
Exclude
Confirm
Required Certainty
Actual Certainty 2.4 = 1.2 x 2
2.0
![Page 55: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/55.jpg)
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
Rule-in, Confirmation Zone High LR+ is better
LR+ > 5
Required LR+ > 𝑷𝒐𝒔𝒕𝒆𝒓𝒊𝒐𝒓 𝑶𝒅𝒅𝒔
𝑷𝒓𝒊𝒐𝒓 𝑶𝒅𝒅𝒔= 𝟓
![Page 56: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/56.jpg)
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
Rule-out, Exclusion Zone: Low LR- is better
LR- < 0.2
Required LR- < 𝑷𝒐𝒔𝒕𝒆𝒓𝒊𝒐𝒓 𝑶𝒅𝒅𝒔
𝑷𝒓𝒊𝒐𝒓 𝑶𝒅𝒅𝒔= 0.2
![Page 57: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/57.jpg)
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
Capabilities > Requirements?
SnNout
SpPin
Required (LR), determined by the customer
actual LR, capabilities
SnNout SpPin
![Page 58: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/58.jpg)
Key Points:
• Accuracy ≠ Usefulness
• Potential Usefulness = LR = f(Sn,Sp)
• Usefulness = Capabilities - Requirements:
– The objective (exclude, confirm)
–Prior uncertainty
–Required certainty
– The Test Impact (actual LR vs required LR)
![Page 59: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/59.jpg)
5 10 15 20 25 30 35
Disease present
No Disease
Setting Test Thresholds
?????
![Page 60: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/60.jpg)
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
ROC curve = set of available Likelihood Ratios
Low LR+ = 1 No impact
High LR+ Confirm
![Page 61: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/61.jpg)
How to select a threshold value
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
![Page 62: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/62.jpg)
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
Diagnostic Zones for Thresholds
SnNout
SpPin Deadly, easily curable disease Pneumonia
Awful treatment, Low chance of cure Cancer
![Page 63: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/63.jpg)
1 – specificity = FPR
0.5 1.0
1.0
0.5
Sen
siti
vity
= T
PR
0.0
Accuracy , Usefulness and Optimality
SnNout
SpPin
Requirements, determined by the customer
![Page 64: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/64.jpg)
• Comparing Tests
– Thresholds are a nuisance
– ROC/AUC facilitates comparisons of diagnostic accuracy
• Using Tests
– Thresholds are required
• Define a test
• Link capabilities and requirements
• Can be set to optimize performance
–Optimum is context dependent
– Depends on error costs
Key Points – Setting Thresholds
![Page 65: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/65.jpg)
Comparing Test Performance
Why do test results differ? 1. True differences 2. False differences
• (bias) • thresholds
3. Random variation (imprecision)
![Page 66: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/66.jpg)
Evaluating Test Accuracy: Ideal vs Realistic Conditions
Consecutive Patients
Index Test
Ideal Realistic
![Page 67: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/67.jpg)
Test Performance Ideal vs Realistic Conditions
1 – specificity = FPR
0.5 1.0 Sen
siti
vity
= T
PR
0.0
Ideal
Realistic
![Page 68: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/68.jpg)
Referral Pattern & Disease Spectrum
Patient with Complaint
General Practitioner
Emergency Dept
![Page 69: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/69.jpg)
Referral Pattern & Disease Spectrum
Patient with Complaint
General Practitioner
Specialist
![Page 70: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/70.jpg)
Effect of Prior Testing Will the index test perform differently?
DRE US *PSA
Patient with
Nausea
*PSA
Diagnosed
*Index Test = the test of interest
![Page 71: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/71.jpg)
Defining a Test: PICCO Sources of Real Differences: Context is Everything
P Population Setting Exclusion/Inclusion criteria Referral pattern Comorbidities Age, Gender
I Index Test Method (in detail) Cutoff Skill level
C Condition Disease of interest
C Comparator (reference test)
Definition of disease
O Outcome measure Diagnostic accuracy Discomfort, adverse events Operational (TAT, Availability, cost, etc)
![Page 72: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/72.jpg)
Comparing Test Performance
Why do test results differ? 1. True differences 2. False differences
• bias • thresholds
3. Random variation (imprecision)
![Page 73: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/73.jpg)
Sources of Bias (Phantom Differences)
• Imperfect gold standard
• Verification Bias
• Indeterminate results
• Others….
![Page 74: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/74.jpg)
Imperfect Gold Standard (Differential verification)
Patients Index Test
Brass Standard
Negative
Positive
X %
X %
![Page 75: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/75.jpg)
Verification bias (Differential sampling)
Patients Index Test
Negative
Positive
X %
Y %
![Page 76: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/76.jpg)
Bias due to indeterminates
Evaluator A: No indeterminates
Evaluator B: Many indeterminates
High sensitivity High specificity
Low sensitivity Low specificity
![Page 77: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/77.jpg)
Indeterminates: Where do these values go?
Gold Standard
Index Test Disease Present
Indeterminate Disease Absent
Positive X
Indeterminate U Y V
Negative Z
![Page 78: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/78.jpg)
Comparing Test Performance
Why do test results differ?
1. True differences
2. False differences (bias)
3. Random variation (imprecision)
![Page 79: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/79.jpg)
Specificity
Understanding Statistical Variation in Studies Meta-Analysis
Outlier?
Average
![Page 80: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/80.jpg)
Comparing tests
Source of Difference Countermeasures
True Differences Complete Reporting PICCO Meta-analysis
False Differences bias Improved Study Design thresholds ROC Curves
Random variation Study design Meta-analysis
![Page 81: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/81.jpg)
Higher Levels of Test Evaluation
Analytical performance
Clinical performance
Clinical effectiveness
Cost effectiveness
Societal Impact
![Page 82: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/82.jpg)
Problems with Test Evaluation
• Potentially useful ≠ Clinically useful
• Potential problems
– Tests are not used properly
– Tests do not change diagnosis
– Tests do not change management
• Tests are not used in isolation
– Incremental value
![Page 83: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/83.jpg)
Clinical Trial Evaluation of Tests
Key Question: Do patients who receive this test have better outcomes?
![Page 84: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/84.jpg)
Tests don’t exist in isolation
• Test Research vs Diagnostic Research
• What is the incremental value of a test?
Test A Test B Test C
Index Test
Patient with Nausea
Index Test
Diagnosed
![Page 85: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/85.jpg)
Tests are often combined
Probability of Disease
Test 1
Test 2
Test n
![Page 86: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/86.jpg)
The acid test
Patients
Usual Tests
Usual Tests
Index Test
Patient Outcomes
Patient Outcomes
Treatment
Treatment
![Page 87: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/87.jpg)
Clinical Trial Evaluation Prostate Screening (PSA)
Event Rate per 1000
Outcomes No Screen
Screen Relative Risk
All cause mortality
200 198 0.99 [0.97-1.01]
Death from prostate CA
8 7 0.88 [0.71-1.09]
Prostate CA diagnosis
44 64 1.46 [1.21-1.77]
![Page 88: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/88.jpg)
Levels of Evaluation
Phase II/III Trial – Explanatory Trial Scientific Perspective Hypothesis: Does this drug affect outcomes? As-Treated Analysis Carefully controlled population, setting Carefully controlled administration and monitoring
Phase III Trial – Pragmatic Trial Policy Perspective Hypothesis: Does prescribing this drug affect outcomes? Intention-to-Treat Analysis Patients seeking treatment for condition Usual conditions
Scientific Test Evaluation Single test Idealized population Expert administration Expert interpretation
Pragmatic Test Evaluation Multiple tests Actual population Usual conditions
Analytical Performance
Discrimination – Ideal (disease vs nondisease)
Discrimination - Real
Patient Outcomes
Cost Effectiveness
Therapeutics Diagnostics
![Page 89: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/89.jpg)
Cost-Effectiveness Plane
Cost
Effectiveness
Lower Cost More
Effective
Higher Cost Less
Effective
Higher Cost More Effective
Lower Cost Less Effective
New Diagnostic Test CER = $10,000/QLY New Cancer Drug
CER = 100,000/QLY
![Page 90: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/90.jpg)
• Many ways to assess performance • Many reasons why studies differ
– Real differences (PICCO) – False differences
• Thresholds • Bias
– Statistical variation
• Progress in Performance Evaluation – Quality of Reporting – Quality of studies – Types of studies
• Educating Clinicians
Summary
Analytical performance Clinical performance Clinical effectiveness
Cost effectiveness Societal Impact
![Page 91: Beyond Sensitivity and Specificity: Testing a Testarup.utah.edu/media/testing_test/Schmidt Testing a Test.pdf · 2015-06-04 · Learning Objectives 1. Learn how to calculate basic](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f045f6f7e708231d40da798/html5/thumbnails/91.jpg)
Testing A Test: Beyond Sensitivity and Specificity