diagnostic testing

Diagnostic TestingDiagnostic Testing

Ethan Cowan, MD, MSDepartment of Emergency Medicine

Jacobi Medical CenterDepartment of Epidemiology and Population Health

Albert Einstein College of Medicine

The Provider DilemmaThe Provider Dilemma

A 26 year old pregnant female presents after twisting her ankle. She has no abdominal or urinary complaints. The nurse sends a UA and uricult dipslide prior to you seeing the patient. What should you do with the results of these tests?

The Provider DilemmaThe Provider Dilemma

Should a provider give antibiotics if either one or both of these tests come back positive?

Why Order a Diagnostic Test?Why Order a Diagnostic Test?

When the diagnosis is uncertain

Incorrect diagnosis leads to clinically significant morbidity or mortality

Diagnostic test result changes management

Test is cost effective

Clinician Thought ProcessClinician Thought Process

Clinician derives patient prior prob. of disease: H & P Literature Experience

“Index of Suspicion” 0% - 100% “Low, Med., High”

Threshold Approach to Threshold Approach to Diagnostic TestingDiagnostic Testing

P < P(-) Dx testing & therapy not indicated P(-) < P < P(+) Dx testing needed prior to therapy P > P(+) Only intervention needed

Pauker and Kassirer, 1980, Gallagher, 1998

Probability of Disease

0% 100%Testing Zone

P(-) P(+)

Threshold Approach to Threshold Approach to Diagnostic TestingDiagnostic Testing

Width of testing zone depends on: Test properties Risk of excess morbidity/mortality attributable to the test Risk/benefit ratio of available therapies for the Dx

Probability of Disease

0% 100%Testing Zone

P(-) P(+)

Pauker and Kassirer, 1980, Gallagher, 1998

Test CharacteristicsTest Characteristics

Reliability Inter observer Intra observer Correlation B&A Plot Simple Agreement Kappa Statistics

Validity Sensitivity Specificity NPV PPV ROC Curves

ReliabilityReliability

The extent to which results obtained with a test are reproducible.

ReliabilityReliability

Not Reliable Reliable

Intra rater reliabilityIntra rater reliability

Extent to which a measure produces the same result at different times for the same subjects

Inter rater reliabilityInter rater reliability

Extent to which a measure produces the same result on each subject regardless of who makes the observation

Correlation (r)Correlation (r)

For continuous data r = 1 perfect r = 0 none

O1 = O2

O1

O2Bland & Altman, 1986

Correlation (r)Correlation (r)

Measures relation strength, not agreement

Problem: even near perfect correlation may indicate significant differences between observations

O1 = O2

r = 0.8

O1

O2Bland & Altman, 1986

Bland & Altman PlotBland & Altman Plot

For continuous data Plot of observation

differences versus the means

Data that are evenly distributed around 0 and are within 2 STDs exhibit good agreement

0

10

-10

O1 – O2

[O1 + O2] / 2

Bland & Altman, 1986

Simple AgreementSimple Agreement

Extent to which two or more raters agree on the classifications of all subjects

% of concordance in the 2 x 2 table (a + d) / N Not ideal, subjects may fall on diagonal by chance

Rater 1Rater 2- + total

- a b a + b + c d c + d

total a + c b + d N

KappaKappa

The proportion of the best possible improvement in agreement beyond chance obtained by the observers

K = (pa – p0)/(1-p0)

Pa = (a+d)/N (prop. of subjects along the main diagonal)

Po = [(a + b)(a+c) + (c+d)(b+d)]/N2 (expected prop.)

Rater 1Rater 2- + total

- a b a + b + c d c + d

total a + c b + d N

Interpreting Kappa ValuesInterpreting Kappa Values

K=1

K > 0.80

0.60 < K < 0.80

0.40 < K < 0.60

0 < K < 0.40

K = 0

K < 0

Perfect

Excellent

Good

Fair

Poor

Chance (pa = p0)

Less than chance

Weighted KappaWeighted Kappa

Used for more than 2 observers or categories Perfect agreement on the main diagonal weighted

more than partial agreement off of it.

Rater 1

Rater 2

1 2 ... C total

1 n11 n12 ... n1C n1.

2 n21 n22 ... n2C n2.

 . .

 . .

 . .

...

... . .

 . .

C nC1 nC2 ... nCC nC.

total n.1 n.2 ... n.C N

ValidityValidity

The degree to which a test correctly diagnoses people as having or not having a condition

Internal Validity External Validity

ValidityValidity

Valid, not reliable Reliable and Valid

Internal ValidityInternal Validity

Performance Characteristics Sensitivity Specificity NPV PPV ROC Curves

2 x 2 Table2 x 2 Table

TP = True Positives

FP = False Positives

Test Result

Disease Status

cases noncases total

+ TP-

positives negatives

total cases noncases NFN

FPTN

TN = True Negatives

FN = False Negatives

Gold StandardGold Standard Definitive test used

to identify cases Example: traditional

agar culture The dipstick and

dipslide are measured against the gold standard

Sensitivity (SN)Sensitivity (SN)

Test Result

Disease Status

cases noncases total

+ TP-

positives negatives

total cases noncases NFN

FPTN

Probability of correctly identifying a true case TP/(TP + FN) = TP/ cases High SN, Negative test result rules out Dx (SnNout)

Sackett & Straus, 1998

Specificity (SP)Specificity (SP)

Test Result

Disease Status

cases noncases total

+ TP-

positives negatives

total cases noncases NFN

FPTN

Probability of correctly identifying a true noncase TN/(TN + FP) = TN/ noncases High SP, Positive test result rules in Dx (SpPin)

Sackett & Straus, 1998

Problems with Problems with Sensitivity and Specificity Sensitivity and Specificity

Remain constant over patient populations But, SN and SP convey how likely a test

result is positive or negative given the patient does or does not have disease

Paradoxical inversion of clinical logic Prior knowledge of disease status obviates

need of the diagnostic test

Gallagher, 1998

Positive Predictive Value (PPV)Positive Predictive Value (PPV)

Test Result

Disease Status

cases noncases total

+ TP-

positives negatives

total cases noncases NFN

FPTN

Probability that a labeled (+) is a true case TP/(TP + FP) = TP/ total positives High SP corresponds to very high PPV (SpPin)

Sackett & Straus, 1998

Negative Predictive Value (NPV)Negative Predictive Value (NPV)

Test Result

Disease Status

cases noncases total

+ TP-

positives negatives

total cases noncases NFN

FPTN

Probability that a labeled (-) is a true noncase TN/(TN + FN) = TP/ total negatives High SN corresponds to very high NPV (SnNout)

Sackett & Straus, 1998

Predictive Value ProblemsPredictive Value Problems

Vulnerable to Disease Prevalence (P) Shifts Do not remain constant over patient populations As P PPV NPV As P PPV NPV

Gallagher, 1998

Flipping a Coin to Dx AMI for Flipping a Coin to Dx AMI for People with Chest PainPeople with Chest Pain

SN = 3 / 6 = 50%SP = 47 / 94 = 50%

AMI No AMI

Heads (+) 3 47 50

Tails (-) 3 47 50

6 94 100

ED AMI Prevalence 6%

PPV= 3 / 50 = 6%NPV = 47 / 50 = 94%

Worster, 2002

Flipping a Coin to Dx AMI for Flipping a Coin to Dx AMI for People with Chest PainPeople with Chest Pain

SN = 45 / 90 = 50%

SP = 5 / 10 = 50%

AMI No AMI

Heads (+) 45 5 50

Tails (-) 45 5 50

90 10 100

CCU AMI Prevalence 90%

PPV= 45 / 50 = 90% NPV = 5 / 50 = 10%

Worster, 2002

Receiver Operator CurveReceiver Operator Curve

Allows consideration of test performance across a range of threshold values

Well suited for continuous variable Dx Tests

1.0

1-Specificity (FPR)

Sensitivity(TPR)

0.00.0 1.0

Receiver Operator CurveReceiver Operator Curve

Avoids the “single cutoff trap”

No Effect

Sepsis

Effect

WBC CountGallagher, 1998

Area Under the Curve (Area Under the Curve (θ) θ)

1-Specificity (FPR)

Sensitivity(TPR)

1.0

0.00.0 1.0

Measure of test accuracy (θ) 0.5 – 0.7 no to low discriminatory power (θ) 0.7 – 0.9 moderate discriminatory power (θ) > 0.9 high discriminatory power

Gryzybowski, 1997

Problem with ROC curvesProblem with ROC curves

Same problems as SN and SP “Reverse Logic”

Mainly used to describe Dx test performance

Appendicitis ExampleAppendicitis Example

Study design: Prospective cohort Gold standard: Pathology report from

appendectomy or CT finding (negatives)

Diagnostic Test: Total WBC

Cardall, 2004

Appy No Appy

CT ScanOR

+

+

- -

Physical Exam

Appendicitis ExampleAppendicitis Example

WBC Appy Not Appy Total

> 10,000 66 89 155

< 10,000 21 98 119

Total 87 187 274

SN 76% (65%-84%)SP 52% (45%-60%)

PPV 42% (35%-51%)NPV 82% (74%-89%)

Cardall, 2004

Appendicitis ExampleAppendicitis Example

Patient WBC: 13,000 Management: Get CT with PO & IV

Contrast

Cardall, 2004

Appy No Appy

CT ScanOR

+

+

- -

Physical Exam

Abdominal CTAbdominal CT

Follow UPFollow UP

CT result: acute appendicitis

Patient taken to OR for appendectomy

But, was WBC necessary?But, was WBC necessary?

Answer given in talk on Likelihood Ratios