National Center for Emerging and Zoonotic Infectious Diseases

HAI Data Internal ValidationSuparna Bagchi, MSPH, DrPH

HAI Data Validation LeadNHSN Protocol and Validation Team

2018 NHSN Annual Training February 26th , 2018

Session Objectives Describe attributes of

– High quality HAI surveillance– Data quality measures– Why does it matter

Consider– How internal data validation can help achieve high data quality

Recommend– How facilities can check their own data

Healthcare Associated Infections (HAI) Surveillance Ongoing Systematic collection Analysis Interpretation Dissemination

Of HAI data that is essential to planning and implementing prevention measures

Simplicity Objectivity Flexibility Data quality Acceptability Sensitivity Positive predictive value Representativeness Timeliness Stability

Quality HAI Surveillance System Requires

Data Quality of HAI Surveillance Reflects

Consistency of data– completeness, timeliness, confidence on your data

Validity– accuracy of data

These surveillance attributes can be achieved by HAI Data Validation

Why Validate?

These are YOUR facility data– Helps assess whether prevention efforts are effective

Ability to hold up during external scrutiny (external validation by State or CMS)– Incomplete or inaccurate surveillance affects payment and/or facility

reputation

You may be surprised at what you find!

Types of HAI Data Validation

Internal Validation Active efforts by a

reporting facility to assure completeness and consistency of NHSN data

Built in as a routine facility process

External Validation Survey and audit process

by external agency to assure accuracy of NHSN surveillance and reporting

Requires additional resources

HAI Data Validation

Internal Validation

ConsistencyData CompletenessTimeliness

External Validation

Data AccuracyImproves

HAI Internal Validation Would Provide

Understanding of systematic weaknesses in HAI reporting

Assurance that surveillance data are of high quality – Complete, timely and accurate

Coordination and partnership building with stakeholders

Confidence in their own data

HAI Internal Validation Toolkit

2018 HAI Internal Validation Toolkit Guidance for planning data quality checks Data quality checklist for review prior to data submission Data quality check tools

– CLABSI/CAUTI Denominator Survey (with key)– Surgical procedures and SSI Surveillance Methods Survey (with key)– LabID Event Surveillance Methods Survey (with key)

Suggestions for Planning Data Quality Checks Annually

– Draft facility level surveillance and validation plan– Recruit partners and update staff training needs – Review facility descriptors, location mapping

Monthly – Determine monthly HAI reporting plan– Review checklist and conduct data quality checks before submission

Daily/Weekly – Spot checks (denominator tracking, surgical procedures)

Medical school affiliationNumber of bedsLocation mapping

Recommended Annual Check

Recommended Annual Check: NHSN Manual

Are staff up to date with NHSN protocol • Review NHSN newsletter, webinars, case studies examples

Annual Checks: Facility Infrastructure

Adequacy of facility infrastructure

– Manual (daily/weekly sampled) or electronic denominator reporting

– EMR systems, access to IT and other support services for planned data checks

– Ability to link laboratory and ADT data

Annual Check: CLABSI & CAUTI Manual Denominator Data

Protocol: daily or weekly sampled denominator data

– Assure staff know correct NHSN procedures and definitions

– Assess denominator data collection practices

– Internal validation annually for one week, each location type, new personnel by concurrent dual assessment to test for compliance

Annual Check: Manual CLABSI/CAUTI Denominators

Protocol: manual, daily or weekly sampling– Are staff counting correctly

• Definition of central line? Which lines are counted – Missing or implausible data

• # patient days > # beds• # catheter days days ># patient days

– Keeping logs of % of days in year• When catheter/central line days not collected• Patient days not collected

Review results with staff !

Data Quality Check: Missing Patient Days

Data Quality: Patient Days > Upper Limits

Upper Limit: Location Beds * 30

Data Quality Check: Patient Days = Device Days = 0

Annual Check: CLABSI & CAUTI Electronic Denominator Data

Electronic denominator estimates are commonly inflated Before you begin: validate e-denominators with concurrent

manual counts * 3months– Counts should match within +/- 5% of manual counts– Work with IT to correct electronic counting problems, otherwise

manual counts should be continued

Current users: conduct spot checks of electronic data to assure continued good performance

Annual Check: SSI Denominator Data (Procedures)

Which NHSN procedures will be reported, inpatient or outpatient, surveillance period (30 or 90 days)

Ensure denominator data are not missed – Identify all data sources: OR records system, hospital discharge

ICD-10-PCS codes – Data linkage of multiple sources reduces missing procedure data or

procedures reported by error

Annual downloads to confirm procedure data from selected days/weeks were not missed during the interval

Common SSI Denominator Reporting Errors

Multiple procedures performed via the same incision on the same trip to operating room– Facility will need to correctly determine which procedure to attribute

the SSI event

Patient returns to operating room for another procedure <24 hours of the first procedure– Correctly apply the 24 hours rule from the SSI protocol

SSI Data Quality Check: BMI Outliers

-- RecordsBMI_val less than 12 2130

Greater than 65 but less than 100 4002Greater or equal to 100 but less than 200 1250

Greater or equal to 200 but less than 1000 2063Greater or equal to 1000 2063

Percent of records with outlier BMI value 0.45%

Recommended Monthly Analysis Checks Analysis

– Generate datasets– Output options– Advanced – Data Quality

NHSN Alerts

SSI Denominator Quality Validation

NHSN SSI Quality Report at Hospital X

Impacts of SSI Quality Checks on Facility-Level Data Facility Procedure Error(s) Correct ion(s) Impact on Facility SSI Data

A COLO Unusually high infect ion incidence Missing 4 months of procedures

Added 127 missing procedures

Reduct ion of 2012 Complex (A/R) SSI SIR from 1.6 to 0.9

B HYST Unusually high infect ion incidence Missing 2 months of procedures

Added 88 missing procedures

Reduct ion of 2012 Complex A/R SSI SIR from 1.1 to 0.9

C COLO 3 procedures missing variables required for risk adjustment

Added missing variables to 3 procedures

Number of expected infect ions based on risk adjustment for the 2012 Complex A/R SSI SIR increased from 0.36 to 0.43

CLABSI/CAUTI Denominator Errors

LabID Reporting: FacWideIN Denominator Issues

Three rows of data entry– Each denominator row should be a sub-set of the row above it– Acute Care or Critical Access hospitals

LabID Reporting: FacWideIN Denominator Issues

Rows 2 and 3 should represent total number of patient days and admissions from all patients in eligible units

Suggestions: LabID Events Reporting Data Quality Checks

Denominators in Rows 2 and 3 are used in the MRSA and CDI SIRs– CDC exploring options to re-label these fields on the denominator form

NHSN has published several newsletter articles about this, and numerous attempts at email outreach (https://www.cdc.gov/nhsn/pdfs/newsletters/nhsn-nl-jun-2017.pdf)

Additional data quality checks: look for facilities that selected “Other” CDI test type – cdiTestMeth = “OTH”; cdiTestMethOth = <free text entry>– Many facilities use the free text entry to indicate a CDI test method that is

already available in the drop-down in NHSN (PCR, DNA amplification, NAAT)– If facilities do not select the option from the drop-down, causes inaccurate

risk adjustment

Review CDI Test Method Selected

Common CLABSI/CAUTI Numerator Errors

Facilities more likely to ‘under-report’ than ‘over-report’

Missed positive blood and urine cultures

Correctly able to identify “present on admission” status

Assigning incorrectly secondary BSI status

Common SSI Numerator Reporting Errors

Depth of the SSI often improperly assigned

Improper use of the infection present at the time of surgery (PATOS) criteria when there is no actual documentation of PATOS

Conversely, not to report a SSI event when PATOS is present

Recommended Case-Ascertainment Checks

Validation of case-ascertainment should include periodically reviewing list of candidate cases

– Use Medical Record Abstraction Tools (MRATS) to identify accuracy of case-ascertainment from a list of candidate cases

– If candidate cases were ‘missed’ investigate why and how to fix it

Daily/Weekly Checks

Review of positive blood and urine cultures

Spot checks of denominator reporting

Determined by facility as per need

Run Longitudinal Data Checks

Review longitudinal trends and assess errors

– Numerators by location and overall

– Denominators by location and overall

– Rates by location and overall

Longitudinal Trends of CLABSI Events, LTAC Ward

Review Checklist of Data Quality Checks Examples

– Missing data (zero patient days)– Implausible data (device days > patient days, BMI > 200) – Complete data (SSI procedures = events)– Outstanding NHSN alerts

Please review these prior to data submission

Data Quality Check Surveys

How to Achieve Numerator Data Completeness ? -- Minimum RequirementCLABSI Review every posit ive blood culture CAUTI Review every posit ive urine cultureSSI • Ident ify and review all post-op pat ients and hospital

readmissions related to infect ions• Review wound cultures but realize culture-based

surveillance missed 50-60% SSI• Daily hospital rounds to ident ify infect ions not result ing in

cultures LabID event FacWideIN

• Review all final lab test results (MRSA cultures,C. diff tests)• Assess lab tests from ERand observat ion locat ions

How to Achieve Denominator Data Completeness ?

-- Minimum RequirementCLABSI/CAUTI • Review presence of central line/ indwelling Foley

catheter and complete collect ion of device daysSSI • Review complete count of procedures based on ICD-

10PCS/CPT codes LabID event FacWideIN

• Review total pat ient admissions/encounters for every locat ion (including ERand observat ion locat ions)

Coordination of Support for IPs

Responsibilities of surveillance and data validation need to be shared across hospital units, services and disciplines

IP needs protected times for HAI prevention activities

Who Can Support IP ?

-- Partner

CLABSI/CAUTI Trained CLABSI/CAUTI monitorsIT to review electronic data downloads

SSI • Surgery/ clinical staff • Surgical ward staff • Medical records

LabID event FacWideIN

Trained LabID event monitors

Confidence In Your Data

Facilities held accountable for using NHSN methods and definitions

Up to date with NHSN surveillance definitions

Apply definitions with confidence every time

Seek assistance for ambiguity

Partnership with State Health Department State HAI Coordinator helps conduct routine internal validation

– Consider allowing state health department access to your facility data

– HAI coordinator runs routine data quality checks for missing and implausible data

– Health Department collaborates with facilities in improving data quality

State Has Access To Facility Data

85

7074

7874 76

15

3026

2226 24

0

20

40

60

80

100

CLABSI CAUTI SSI-COLO SSI-HYST CDI MRSA

Yes No

Perc

ent

States That Have Conducted Data Quality Checks:6 Months of 2015 NHSN Data

72

5663 65 63

57

28

4437 35 37

43

0

20

40

60

80

100

CLABSI CAUTI SSI-COLO SSI-HYST CDI MRSA

YesNo

Perc

ent

State Contact Facilities For Reporting Errors ?

82%

18%

YesNo

What Should Trigger Data Quality Checks

New patient care locations: accurately mapped?

New or modified electronic medical record systems

Unusual high incidence of events – Was denominator reporting complete? – Were all risk adjustment variable entered correctly – Was there a change in testing method (e.g., CDI testing method)

Summary

Credible data is vital to HAI prevention

In the era of “publicly looking good” ongoing validation is the key to improvement in prevention practices

Regular internal validation will improve results in external validation

For more information, contact CDC1-800-CDC-INFO (232-4636)TTY: 1-888-232-6348 www.cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

Thank You !

Questions