1

Ann Versporten,

Ingrid Morales, Carl Suetens

IPH, wednesday seminar: May 7, 2003

Scientific Institute of Public Health

Data validation study of the National surveillance of

nosocomial infections in intensive care units

2

Overview• Background: overview national surveillance ICU• Reasons for validation• Validation study

– Aims– Methods– Results

• Pneumonia• Bacteraemia

– Discussion– Conclusions– Recommendations

3

Background: National surveillance ICU

• 1996: Start National Surveillance of Hospital Infections (NSIH) : intensive care component (Pneumonia & Bacteraemia) – HELICS-based protocol (Hospitals in Europe

link for Infection Control through Surveillance)– patient-based surveillance: 1 file by patient, +

infection file if ICU-acquired PN or BAC– Nosocomial: infection acquired during hospital

stay (admitted >48h in ICU)

4

Background: National surveillance ICU

• Objective: to follow-up nosocomial-infection rates

• Risk-adjusted infection rates are used as external benchmarks for comparison purposes

5

Methods: Data collection for ICU surveillance

1. Data at admission

2. Day-by-day e.g. central venous catheter, mechanical

ventilation, antibiotic use

3. Infection data e.g. diagnostic criteria of PN, origin of BSI

4. Data at discharge

6

Reasons for validation

• Assessment of the validity of the findings

• Need to evaluate the accuracy of infection data reported to the NSIH program

7

Validation study

8

Main aim

• Validate reported ICU-surveillance data (ICU protocol: PN & Bac) against a reference gold standard

• Evaluate the accuracy of all data reported to the surveillance

• Evaluate the credibility of the surveillance

9

Specific aims

• Exhaustivity (completeness) denominator

• Sensitivity: probability of reporting a true PN & Bac to the ICU-surveillance

• Specificity: probability of reporting a PN & Bac as negative to the ICU-surveillance if the disease is truly absent

• Positive predictive value

• Negative predictive value

11

Methods - 1

• Sampling of hospitals: Systematic sampling of 45 hospitals on the

base of a list of hospital-trimesters

(ICU participation period 01/01/1997 – 31/12/1999)

• Replacement: later period accepted

• Informed consent, voluntary participation

• Retrospective chart review methodology

12

Methods - 2: Research program

Sampling of patient files: All reported PN+ & Bac+ (from surv.) All records with a positive hemoculture reported

on a laboratorium list (for all admitted patients on ICU) (estimation false-neg Bac)

A 20% random sample of the negative files (estimation of false-neg PN)

Estimation of exhaustivity of denominator on the base of administrative lists of ICU-admissions

13

Methods - 3• Calculation Se, Sp and Predictive values

“gold standard” = research team

• Trained data collectors (IPH) Application protocol definitions

validation: uniform & standardised evaluation = blind discrepant infections: reviewed by other

colleague

• Confidential & anonymous treatment of patient data

14

Methods - 4

• National results

• No individual hospital results, only discussion at end validation proccess Quality of dataQuestions

15

Results - 1

• 563 investigated patient files in analysis: pts staying >24h in ICU (23 hospitals)

• Infections reported by hospitals to surveillance: 147 Pneumonia 49 Bacteraemia

• Type of ICU: 91% polyvalent• Size of ICU: mean 10 beds • Length of stay: median = 4,7 days

16

Results - 2

• Exhaustivity of denominators:– For all patients staying >24h in ICU

72,8%

– For all patients staying >48h in ICU81,2%

17

Results - 3: Pneumonia

(106/133)*147=117.2(24/430)*1843=102.9

Validation+ - Total

Surv. + 117 30 147- 103 1740 1843

Total 220 1770 1990

Validation+ - Totaal

Surv. + 106 27 133- 24 406 430

Total 130 433 563

All PN inf.file &/or bdb Freq %

1 147 7.392 1843 92.61

Total 1990 100

Results of validation study for PN (inf. file &/or dbd) Results from surveillance for PN (inf. file &/or dbd)

Results applied on total sample(proportional balancing to files not been validated

18

Results - 4: Bacteraemia

Validation+ - Total

Surv. + 32 17 49- 22 1919 1941

Total 54 1936 1990

Validation+ - Total

Surv. + 32 12 44- 22 497 519

Total 54 509 563

All Bac inf.file &/or bdb Freq %

1 49 2.462 1941 97.54

Total 1990 100

Results of validation study for Bac (inf. file &/or dbd) Results from surveillance for Bac (inf. file &/or dbd)

Results applied on total sample(proportional balancing to files not been validated

19

Results: SE & SP

Se % (95% CI) Sp % (95% CI)

Pneumonia

Infection file 32,7 (25,2-41,2) 98,5 (97,4-99,2)

Inf.file &/or dbd 53,2 (43,5-62,7) 98,5 (97,4-99,0)

Bacteraemia

Infection file 48,1 (29,2-67,6) 99,3 (98,5-99,7)

Inf.file &/or dbd 59,3 (39,0-76,9) 99,1 (98,2-99,6)

20

Results: predictive values

PPV (%)(CI) NPV (%)(CI)

Pneumonia

Infection file 78,6 (65,6-87,9) 85,9 (83,5-87,9)

Inf.file &/or dbd 79,6 (68,3-87,8) 88,9 (86,8-90,8)

Bacteraemia

Infection file 65,0 (40,9-83,7) 97,3 (96,0-98,2)

Inf.file &/or dbd 65,3 (43,6-82,4) 97,3 (96,0-98,2)

21

Discussion - 1

• Exhaustivity denominator: improvement possible – risk of bias, e.g. if only high risk patients included

• Pneumonia: low Se., good Sp.

• Bacteraemia: low Se., good Sp.

22

Discussion – 2

• Possible reasons for lack of sensitivity– 30% of the results originate from 1997 (start

surv. NI in ICU). – 50% of the collected data correspond with the 3

first surveillance trimesters that hospitals participated to our ICU surveillance. = Explanation of lack of accuracy in the

interpretation of the protocol ?

23

Who are those missed patients ??

Why are there so many false negative Pneumonias ?

24

Characteristics false negative PN

Pneumonia N % mort.

mean length of

stay (days)

n ventilation

daysmean

SAPS II

median PN Risk Score

% with micro-org.

% pts with >=1 (other)

missing value

mean stay post infection

(days)

True + PN 93 24.7 20.0 7.1 42.6 47 84 8.6 13.9

False - PN 23 30.4 13.6 8.9 35.1 41 70 21.7 9.3

True - PN 1626 6.7 5.6 1.7 31.3 19 - 15.7 -

28

Factors influencing the Se. & Sp. of the infection data

• Who collects data ?

• Who decides whether a PN should be reported or not ?

• Criteria of bloodculture?

• Adherence to protocol definitions

• Degree of workload (ratio pat.-staff)

• Size of hospital

• …

29

Conclusions

• Exhaustivity varies for each hospital, but remains satisfactory in general

• Bac more accurately reported than PN (Se)

• Seldomly infections reported which were not a nosocomial infection (Sp)

• Absence of a gold standard ! (problem for diagnostic of PN)

30

Conclusions (next)

• Establishing Se & Sp only possible at the end of validation studyPreliminary conclusions:

Sensitivity rather low (identification of a NI through surveillance)Specificity is high (% files truly classified as non-NI)

Low Se has also been reported by the CDC: “The data collectors detected over 2,5 times as many PN, ..” (Emori, Edwards, et al. 1998)

31

Recommendations

• Training of professionals in charge of surveillance (Ehrenkranz, Shultz, et al. 1995)

case definitions (e.g. PN-diagnostic: use of micro-biologic reports & AB-administration)

surveillance-methods

• Simplification of protocol

• Development of electronic surveillance

32

Recommendations (next)

• Validation on continuous basis Training on the field Optimalisation contacts IPH / hospitals