teaching about diagnosis tom sensky teaching ebm/ebmh st hughs college oxford september 2005

TEACHING ABOUT DIAGNOSIS

Tom Sensky

Teaching EBM/EBMH

St Hugh’s College OxfordSeptember 2005

BY THE END OF THIS SESSION,YOU SHOULD BE ABLE TO ….

• describe and illustrate key measures of diagnostic test performance

• describe some less commonly quoted measures of diagnostic test performance

• represent diagnostic test performance in at least four different ways (five if time allows!)

METHOD 1: NATURAL FREQUENCIES GRID

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

Assume that the prevalence of the disease is 4%

Assume that of the 4 people with the disease, 3 are picked up by the test

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

Assume that of the test is positive for a further 7 people who don’t have the disease

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

The remainder of the sample are negative on the test

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

SENSITIVITY

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

•SENSITIVITY is the proportion of people with the disease correctly identified by the test

•It measures the proportion of false NEGATIVES

SENSITIVITY

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

In this case, sensitivity is ¾ or 75%

SPECIFICITY

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

•SPECIFICITY is the proportion of people without the disease correctly identified by the test

•It measures the proportion of false POSITIVES

SPECIFICITY

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

In this case, specificity is (96-7)/96 or 93%

If someone is positive on the test, what are the chances that he/she has the disease?

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

•Probability = 3/10 = 30%•This is the POSITIVE PREDICTIVE VALUE (the value of the test in predicting a positive result)

If someone is negative on the test, what are the chances that he/she does not have the disease?

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

Person without the disease

Person with the disease

Person who tests positive

Person who tests negative

True positive on the test

False positive on the test

True negative on the test

False negative on the test

•Probability = 89/90 = 99%•This is the NEGATIVE PREDICTIVE VALUE (the value of the test in predicting a negative result)

SENSITIVITY, SPECIFICITY AND PREDICTIVE VALUES

• For sensitivity and specificity, the reference variable (‘denominator) is the DISEASE

• For predictive value, the reference variable (‘denominator’) is the TEST

METHOD 2: NATURAL FREQUENCIES TREE

Population

100

IN EVERY 100 PEOPLE, 4 WILL HAVE THE DISEASE

Disease +

4

Disease -

96

Population

100

If these 100 people are representative of the population at risk, the assessed rate of those with the

disease (4%) represents the PREVALENCE of the disease – it can also be considered the PRE-TEST

PROBABILITY of having the disease

OF THE 4 PEOPLE WITH THE DISEASE, THE TEST WILL DETECT 3

Disease +

4

Disease -

96

Test +

3

Test -

1

Population

100

In other words, the sensitivity is

75%

AMONG THE 96 PEOPLE WITHOUT THE DISEASE, 7 WILL TEST POSITIVE

Disease +

4

Disease -

96

Test +

7

Test -

89

Test +

3

Test -

1

Population

100

In other words, the specificity is

93%

POSITIVEPREDICTIVE

VALUE = 30%

AMONG THOSE WHO TEST POSITIVE, 3 IN 10 WILL ACTUALLY HAVE THE DISEASE

Disease +

4

Disease -

96

Test +

7

Test -

89

Test +

3

Test -

1

Population

100

This is also the POST-TEST PROB- ABILITY of having

the disease

NEGATIVEPREDICTIVEVALUE = 99%

AMONG THOSE WHO TEST NEGATIVE, 89 OF 90 WILL NOT HAVE THE DISEASE

Disease +

4

Disease -

96

Test +

7

Test -

89

Test +

3

Test -

1

Population

100

CONVERSELY, IF SOMEONE TESTS NEGATIVE, THE CHANCE OF HAVING THE DISEASE IS ONLY 1 IN 90

Disease +

4

Disease -

96

Test +

7

Test -

89

Test +

3

Test -

1

Population

100

PREDICTIVE VALUES AND CHANGING PREVALENCE

Disease +

4

Disease -

996

Population

1000

Prevalence reduced by an order of magnitude from 4%

to 0.4%

PREDICTIVE VALUE AND CHANGING PREVALENCE

Disease +

4

Disease -

996

Test +

70

Test -

926

Test +

3

Test -

1

Population

1000

Sensitivity and Specificity

unchanged

POSITIVEPREDICTIVEVALUE = 4%

POSITIVE PREDICTIVE VALUE AT LOW PREVALENCE

Disease +

4

Disease -

996

Test +

70

Test -

926

Test +

3

Test -

1

Population

1000

Previously, PPV was 30%

NEGATIVEPREDICTIVEVALUE >99%

NEGATIVE PREDICTIVE VALUE AT LOW PREVALENCE

Disease +

4

Disease -

996

Test +

70

Test -

926

Test +

3

Test -

1

Population

1000

Previously, NPV was 99%

PREDICTION OF LOW PREVALENCE EVENTS

• Even highly specific tests, when applied to low prevalence events, yield a high number of false positive results

• Because of this, under such circumstances, the Positive Predictive Value of a test is low

• However, this has much less influence on the Negative Predictive Value

RELATIONSHIP BETWEEN PREVALENCE AND PREDICTIVE VALUE

0

0.2

0.4

0.6

0.8

1

0.05 0.2 0.4 0.6 0.8 0.95

Pre-test Probability (Prevalence)

Pre

dictive V

alu

e

PPVNPV

Based on a test with 90% sensitivity and 82% specificity

Difference between PPV and

NPV relatively small

Difference between PPV and

NPV relatively large

RELATIONSHIP BETWEEN PREVALENCE AND PREDICTIVE VALUE

Based on a test with 75% sensitivity and 93% specificity

Prevalence

Pre

dic

tive V

alu

e

PERFORMANCE OF A TEST WITH CHANGING PREVALENCE

A : Sensitivity = Specificity = 0.9LR+ = 9.0

B : Sensitivity = Specificity = 0.7LR+ = 3.0

C : Sensitivity = Specificity = 0.5LR+ = 1.0

PO

ST-T

ES

T P

RO

BA

BIL

ITY

LIKELIHOOD

Disease +

4

Test +

3

Test -

1

Population

100

The likelihood that someone with the disease will have a positive test is ¾ or 75%This is the same as the sensitivity

LIKELIHOOD II

Disease -

96

Test +

7

Test -

89

Population

100

The likelihood that someone without the disease will have a positive test is 7/96 or 7%This is the same as the (1-specificity)

LIKELIHOOD RATIO

LIKELIHOOD OF POSITIVE TEST IN THE ABSENCE OF THE DISEASE

SENSITIVITY

1- SPECIFICITY= = 10.7

LIKELIHOOD OF POSITIVE TEST GIVEN THE DISEASE

=LIKELIHOOD RATIO

A Likelihood Ratio of 1.0 indicates an uninformative test (occurs when sensitivity and specificity are both

50%)

The higher the Likelihood Ratio, the better the test (other factors being equal)

0.75

0.07=

METHOD 3: ‘TRADITIONAL’ 2x2 TABLES

SENSITIVITY

SENSITIVITY

The proportion of people with the diagnosis (N=4) who are correctly identified (N=3)

Sensitivity = a/(a+c) = 3/4 = 75%

FALSE NEGATIVES

SPECIFICITY

SPECIFICITY

The proportion of people without the diagnosis (N=96) who are correctly identified (N=89)

Specificity = d/(b+d) = 89/96 = 93%

FALSE POSITIVES

PRE-TEST ODDS

In the sample as a whole, the odds of having the disease are 4 to 96 or 4% (the PRE-TEST ODDS)

POST-TEST ODDS

In those who score positive on the test, the odds of having the disease are 3 to 7 or 43% (the POST-TEST ODDS)

In the sample as a whole, the odds of having the disease are 4 to 96 or 4% (the PRE-TEST ODDS)

POST-TEST ODDS

In those who score positive on the test, the odds of having the disease are 3 to 7 or 43% (the POST-TEST ODDS)

In the sample as a whole, the odds of having the disease are 4 to 96 or 4% (the PRE-TEST ODDS)

In those who score negative on the test, the odds of having the disease are 1 to 89 or approximately 1%

DIAGNOSTIC ODDS RATIO

The Diagnostic Odds Ratio is the ratio of odds of having the diagnosis given a positive test to those of having the diagnosis given a negative test

2.38011.0429.089

17

3DOR

Potentially useful as an overall summary measure, but only in conjunction with other measures (LR, sensitivity, specificity)

BAYES THEOREM

POST-TEST ODDS =

LIKELIHOOD RATIO x PRE-TEST ODDS

LIKELIHOOD RATIO AND PRE- AND POST-TEST PROBABILITIES

For a given test with a given likelihood ratio, the post-test probability will depend on the pre-test probability (that is, the prevalence of the condition in the sample being assessed)

SENSITIVITY ANALYSIS OF A DIAGNOSTIC TEST

Value

95% CI

Pre-test probabilit

y35% 26% to 44%

SENSITIVITY ANALYSIS OF A DIAGNOSTIC TEST

Applying the 95% confidence intervals above to the nomogram, the post-test probability is likely to lie in the range 55-85%

Value

95% CI

Pre-test probabilit

y35% 26% to 44%

Likelihood ratio

5.0 3.0 to 8.5

RECEIVER OPERATING CHARACTERISTIC CURVE

Overall shape is predicted by the reciprocal relationship between sensitivity and specificity

The closer the curve gets to Sensitivity=1 and Specificity=1, the better the overall performance of the test

The diagonal line (representing Sensitivity=0.5 and Specificity=0.5) represents performance no better than chance

Hence the area under the curve gives a measure of the test’s performance

FALSE POSITIVE RATE (1-Specificity)

TR

UE P

OS

ITIV

E R

ATE

(Sen

sit

ivit

y)

0

100

1-Specificity

Sensi

tivi

ty

AREA UNDER ROC CURVES

0

100

1-Specificity

Sensi

tivi

ty Sensitivity and specificity both 100% - TEST PERFECT

Sensitivity and specificity both 50% - TEST USELESS

AREA=1.0

AREA=0.5The area under a ROC curve will be between 0.5 and 1.0

0

100

1-Specificity

Sensi

tivi

ty

AREA UNDER ROC CURVES

Area = 0.7 (between 0.5 and

1.0)

•Consider (hypothetically) two patients drawn randomly from the DISEASE+ and DISEASE- groups respectively

• If the test is used to guess which patient is from the DISEASE+ group, it will be right 70% of the time

APPLYING A DIAGNOSTIC TEST IN DIFFERENT SETTINGS

• The Positive Predictive Value of a test will vary (according to the prevalence of the condition in the chosen setting)

• Sensitivity and Specificity are usually considered properties of the test rather than the setting, and are therefore usually considered to remain constant

• However, sensitivity and specificity are likely to be influenced by complexity of differential diagnoses and a multitude of other factors (cf spectrum bias)

RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE

0102030405060708090

100

0 20 40 601-Specificity

Sensi

tivit

y

ACAT

MC

This study compared the performance of a dementia screening test in a community sample (ACAT) and a memory clinic sample (MC)

Flicker L, Loguidice D, Carlin JB, Ames D. The predictive value of dementia screening instruments in clinical populations. International Journal of Geriatric Psychiatry 1997 ; 12 : 203-209

METHOD 4: A TEST WITH NORMALLY DISTRIBUTED VALUES

Negative Positive

Degree of ‘positivity’ on test

% o

f G

rou

p

DISEASED

NON-DESEASED

Test cut-off

Assessing the performance of the test assumes that these two

distributions remain constant. However, each

of them will vary (particularly through spectrum or selection

bias)

CASESNON-CASES

PERFORMANCE OF A DIAGNOSTIC TEST

Negative Positive

Degree of ‘positivity’ on test

% o

f G

rou

p

DISEASED

NON-DESEASED

Test cut-offFALSE

NEGATIVESFALSE

POSITIVES

MINIMISING FALSE NEGATIVES: A SENSITIVE TEST

Negative Positive

Degree of ‘positivity’ on test

% o

f G

rou

p

DISEASED

NON-DESEASED

Test cut-off

Cut-off shifted to minimise false negatives ie to optimise sensitivity

CONSEQUENCES:

- Specificity reduced

- A Negative result from a seNsitive test rules out the diagnosis - snNout

CASESNON-CASES

MINIMISING FALSE POSITIVES: A SPECIFIC TEST

Negative Positive

Degree of ‘positivity’ on test

% o

f G

rou

p

DISEASED

NON-DESEASED

Test cut-off

Cut-off shifted to minimise false positives ie to optimise specificity

CONSEQUENCES:

- Sensitivity reduced

- A Positive result from a sPecific test rules in the diagnosis - spPin

NON-CASESCASES

METHOD 5: USING SCALES WITH DIFFERENT CUT-OFFS

-20 -15 -10 -5 0 5 10 15 20 25

PATIENTS

>24

24

23

22

21

20

19

18

<18

TRUE

POSITIV

ES

TRUE

NEGATI

VES

FALS

E

POSITIV

ES

FALS

E

NEGATI

VES

B

DC

A

Sensitivity = A/A+C

Specificity = D/B+D

MM

SE S

core

Chosen cut-off

NON-CASESCASES

INCREASING SENSITIVITY

-20 -15 -10 -5 0 5 10 15 20 25

PATIENTS

>24

24

23

22

21

20

19

18

<18

TRUEPOSITIVES

FALSENEGATIVES

B

DC

A

Sensitivity = A/A+C

Specificity = D/B+D

MM

SE S

core

In a seNsitive test, false Negatives are minimised

A negative result from a sensitive test rules out the diagnosis (snNnout)

NON-CASESCASES

INCREASING SPECIFICITY

-20 -15 -10 -5 0 5 10 15 20 25

PATIENTS

>24

24

23

22

21

20

19

18

<18

TRUENEGATIVES

FALSEPOSITIVES

B

DC

A

Sensitivity = A/A+C

Specificity = D/B+D

MM

SE S

core

In a sPecific test, false Positives are minimised

A positive result from a specific test rules in the diagnosis (spPin)

KEY REFERENCES

Sedlmeier P and Gigerenzer G. Teaching Bayesian reasoning in less than two hours. Journal of Experimental Psychology: General. 130 (3):380-400, 2001.

Knotternus JA (ed). The Evidence Base of Clinical Diagnosis. London: BMJ Books, 2002.

Sackett DL, Haynes RB, Guyatt G, and Tugwell P. Clinical Epidemiology : A Basic Science for Clinical Medicine. Boston, Mass: Little, Brown & Co, 1991.

Loong TW. Understanding sensitivity and specificity with the right side of the brain. BMJ 2003: 327: 716-19.