The cross sectional study design. Population

and pre-test. Probability (participants). Index

test. Target condition. Reference Standard

Mirella Fraquelli U.O. Gastroenterologia 2

Fondazione IRCCS “Cà Granda” Ospedale Maggiore Policlinico, Milano

Oulines

• Definition and aim of diagnostic accuracy test study

• Key components:

Study design

Population

Index test

Reference standard

Diagnostic test accuracy is the ability of

a test to distinguish between patients and

subjects without the target condition

Target condition: a particular disease,

disease stage or health status that should

prompt clinical action (such as the initiation

modification or termination of treatment)

Analytical diagnostic reasoning

Initial level of probability of a diagnostic hypothesis

(Epidemiology, medical history, physical examination, laboratory etc.)

PRE-TEST PROBABILITY

INDEX TEST RESULT

Variation of the level of probability

POST-TEST PROBABILITY

Level of certainty necessary for a therapeutic decision

- +

Objective of a diagnostic test accuracy study

To assess how the index test change the probability of the disease

LR+ 12

LR- 0.6

10 20 40 60 80 100

10

2

0 4

0 6

0 8

0

9

0 1

00

Test

Information

Pre-test

probability

Po

sr-

test p

rob

ab

ility

Ultrasound scan in the detection of

hepatocellular carcinoma

Colli et al. Am J Gastroenterol 2007

Analytical diagnostic reasoning

The acceptable level of UNCERTAINTY depends on the penalty for being wrong.

For my single patient

it will be better to be treated if false positive or not treated if false negative?

How can I transfer the results obtained from

diagnostic studies to my single patient ?

Internal validity

External validity

Internal validity

Correct study design

Ideal experimental conditions

Data homogeneity

Reduced heterogeneity

Data precision and

repeatability

External validity

Clinically relevant context

No center selection

No patients selection

Co-morbidity

Data transferability

Diagnostic test accuracy studies

1) Critical appraisal of available diagnostic literature by assessing:

- methodological quality (INTERNAL VALIDITY)

- transferability (EXTERNAL VALIDITY)

- reporting (correct expression of diagnostic estimates)

(STARD INITIATIVE)

2) Match the characteristic of my patient to that of those reported in the literature as measures of accuracy may vary across patient groups

Factors affecting the transferability of data derived from diagnostic studies to a single patient

Population

• Diagnostic test validity and generalizability must be evaluated in a clinically relevant population

• An optimal study includes a broad spectrum of persons who would undergo the test in a clinical setting

• However, test performances can varies across population subgroups according to patient characteristics

“Relevant” spectrum of patients

TE

Liver biopsy

Liver biopsy

TP

FP

FN

TN

Basic design of diagnostic accuracy studies: Prospective,

blinded cross classification of test and reference standard in a clinical relevant setting

Study design

Diagnostic patient -control study

Test parameter

Healthy volunteers

%

Very sick individuals

Test threshold

Spectrum effects: evaluation of two very different populations

the healthiest the sickest

SPECTRUM BIAS

Diagnostic cross sectional study

Test parameter

Patients without disease

%

Patients with disease

Test threshold

Spectrum effects: evaluation of representative populations

SPECTRUM VARIATION

Severity of the disease

co-morbidities

Severity of the disease

co-morbidities

Severity of the disease

co-morbidities

Index test

• Validity. A test is valid when it measures what it’s supposed to

• Reliability: A test is reliable when it yields consistent result.

Repeatability and reproducibility ( Inter- and intra-observer

variability)

• Possible sources of danger and costs for unnecessary

procedures

(e.g. direct harm of radiation exposure : 2% of cancers may be

attributed to radiation exposure during CT scanning).

Lauer NEJM 2009; 841-43 Brenner 2007; 357: 2277-84

• Triage

• Add on

• Replacement

EACH SITUATION MAY REQUIRE DIFFERENT TEST

FEATURES!

Remember the purpose of your test !

Existing situation Replacement Triage Add-on

Population Population Population Population

Initial test Initial test Initial test New test

New test

New test

Existing test

Existing test Existing test +

+

+

+ +

+

- -

-

-

-

- Bossuyt et al. BMJ 2006;332:1089-1092

Existing situation Replacement Triage Add-on

Population Population Population Population

Initial test Initial test Initial test New test

New test

New test

Existing test

Existing test Existing test +

+

+

+ +

+

- -

-

-

-

- Bossuyt et al. BMJ 2006;332:1089-1092

Existing situation Replacement Triage Add-on

Population Population Population Population

Initial test Initial test Initial test New test

New test

New test

Existing test

Existing test Existing test +

+

+

+ +

+

- -

-

-

-

- Bossuyt et al. BMJ 2006;332:1089-1092

Existing situation Replacement Triage Add-on

Population Population Population Population

Initial test Initial test Initial test New test

New test

New test

Existing test

Existing test Existing test +

+

+

+ +

+

- -

-

-

-

- Bossuyt et al. BMJ 2006;332:1089-1092

The Reference Standard

The procedure (or test) that is used

to define the true state of the patient

The Reference Standard is generally more invasive and expensive than the index test

The Reference standard to establish the final diagnosis should be applied for all included patients

Index Test

+ -

Reference Standard

TP FP FN TN

If Reference Standard + ≡ Disease +

+ - - +

The reference standard should accurately reflect the true state of the patient, but is usually imperfect.

Its sensitivity and specificity are < 100%

Reference standard + ≠ Target Disease +

From the GOLD STANDARD

to the REFERENCE STANDARD

Pos

A agreement

B Cases detected only by

the Index Test

Neg

C Cases detected only by

the RS

D agreement

IND

EX

T

ES

T

REFERENCE STANDARD

Pos Neg

Index Test more

specific

Index Test

more

sensitive

Glasziou Ann Intern Med 2008; 149:816

Disagreements between Reference Standard and Index Test

Imperfect

reference standard

vs true disease

status

Index test

Trade off Sensitivity/Specificity

Specificity

Sensitivity

Underestimates specificity

Liver biopsy in diagnosing hepatic fibrosis

Broad histopathologic criteria to diagnose colon dysplasia

Underestimates sensitivity

REFERENCE STANDARD: SENSITIVITY < 100 %

INDEX TEST SPECIFICITY

CAN BE UNDERESTIMATED

REFERENCE STANDARD: SPECIFICITY < 100 %

INDEX TEST SENSITIVITY

CAN BE UNDERESTIMATED