evaluation of surveillance systems günter pfaff 2009/10 / viviane bremer 2008 / preben aavitsland /...
TRANSCRIPT
Evaluation of surveillance systems
Günter Pfaff 2009/10 / Viviane Bremer 2008 / Preben Aavitsland / FETP Canada
Günter Pfaff
17th EPIET Introductory CourseLazareto, Menorca, Spain
September – October, 2011
The surveillance loop
Health Care System Public Health Authority
Event Data
InformationIntervention(Feedback)
Reporting
Analysis & Interpretation
Decision
Importance of evaluation
Obligation Does the system deliver?
Credibility of public health service
In reality Often neglected
Basis for improvements
Learning process EPIET training objective
”Do not create one until you have evaluated one”
Detect trends? Epidemics?Provide estimates of morbidity and mortality? Identify risk factors?Stimulate epidemiologic research?Assess effects of control measuresLead to improved clinical practice?Lead to new/improved control measures?Lead to better advocacy and increased
funding?
Does the surveillance system…
Simplicity Flexibility Acceptability Data quality Sensitivity and Predictive
value positive (PvP) Capture-recapture
Representativeness Timeliness
Criteria to look at
CDC guidelines
SimplicityAs simple as possible while meeting the objectives Structure
Information needed Number and type of sources Training needs Number of information users
Functionality Data transmission System maintenance Data analysis Information dissemination
Components of system
• Population under surveillance
• Period of data collection
• Type of information collected
• Data source
• Data transfer
• Data management and storage
• Data analysis: how often, by whom, how
• Dissemination: how often, to whom, how
Confidentiality, security
Flowchart (HIV in Norway)
Referencelaboratory
Primary HIV reporting form,Blood sample for HIV test laboratory part 1
Lab report and HIV reporting form
HIV reporting form, part 2HIV infection Primary care (Prompting if necessary) National Institute
physician of Public Health
AIDS reporting formAIDS Hospital physician Semiannual check
Oral informationDeath, emigration Semiannual check
Patient
Flexibility
Ability of the system to accommodate changes
New event to follow-up New data about an event New sources of information
Acceptability
Willingness to participate in the system Participation (%) of sources Refusal (%) Completeness of report forms Timeliness of reporting
Acceptability Factors influencing the willingness to
participate Public health importance Recognition of individual contribution Responsiveness to comments/suggestions Time burden Legal requirements Legal restrictions
Data quality
Completeness• Proportion of
blank / unknown responses
• Simple counting
Validity• True data?
• Comparison Records inspection Patient interviews ...
Completeness of informationInformation
Total Total
records records
No. No. (%) No. No. (%)
Person
Name 703 703 (100) na
Birth date 703 703 (100) na
Birth month and year 703 703 (100) 1491 1489 (100)
Sex 703 703 (100) 1491 1491 (100)
Municipality of residence at HIV-diagnosis 703 703 (100) 1491 1479 (99)
Country of birth 703 703 (100) 1491 1489 (100)
If not Norway
Reason for stay in Norway 109 100 (92) 592 551 (93)
Length of stay in Norway at HIV-diagnosis 109 62 (57) 592 352 (59)
Place
Infection acquired in Norway or abroad 703 334 (48) 1491 998 (67)
Cases acquired abroad
Country where infection was acquired 196 171 (87) 665 606 (91)
AIDS cases HIV cases without AIDS
Records with
item filled in
Records with
item filled in
Exposed
Clinical specimen
Symptoms
Pos. specimen
Infected
Seek medical attention
Report
Sensitivity
= reported true cases total true cases
= proportion of true cases detected
Disease
Notified
+
+ -
Total sick Total not sick
Total not notified
Totalnotified
True -
False +
False -
True +
-
Sensitivity = True + / Total sick
Specificity = True - / Total Not sick
PVP = True+ / Total notified
Sensitivity
-
-
Sensitivity versus specificity
The tiered system: confirmed, probable, possible
Frequent "false-positive" reports Inappropriate follow-up of non-cases Incorrect identification of epidemics
Wastage of resourcesInappropriate public concern (credibility)
Consequences of low PvP
Measuring sensitivity
• Find total true cases from other data
sources medical records
disease registers
special studies
• Capture-recapture study
Capture-recapture
• Used for counting total number of individuals in population using two or more incomplete lists
• Originally used in wildlife counting(birds, polar bears, wild salmon…)
Uses in epidemiology
• Estimate prevalence or incidence from incomplete sources
• Evaluate completeness of a surveillance system
Principles
• Two/more sources of cases with disease Lists, registries, observations, samples
• Estimate total number in the source population (captured and uncaptured) from the numbers of captured in each capture
Assumptions
1. The population is closed No change during the investigation
2. Individuals captured on both occasions can be matched No loss of tags
3. For each sample, each individual has the same chance of being included Same catchability
4. Capture in the second sample is independent of capture in the first The two samples are independent, pYZ = pY pZ
Seaworld Oberhausen, August 2010
Daddy, how many fish are in the aquarium?
Your options as a scientist
• Don‘t answer => Expect repeat question
• Answer something => „How do you know?“
• Consult an expert
• Estimate yourself
Meet the expert - „Pulpo Paul“• Has nine brains and three hearts
• Managed to predict all German games during the 2010 Football World Cup right
• Predicted accurately the finale Netherlands-Spain
Binomial distributions
only
http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes
Two-source model
Source Z
Source Y
b a c
x=?
N=?
N= a + b + c + x
Z1
Y1
Two-source analysis
Yes No
Yes a b Z1 = a + b
No c x
Y1 = a + c N = a + b + c + x
Source Y
Source Z
N = Y1 Z1 / a
Sensitivity of Y Ysn = Y/N = (a+c)/NSensitivity of Z Zsn = Z/N = (a+b)/N
How many persons are in the EPIET 2011 Introductory Course?Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“
How many persons are in the EPIET 2011 Introductory Course?Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“, n=33
3
4 4
4
5
4
2
3
3
1
Hand does not meet our case
definition
This is our first view
How many persons are in the EPIET 2011 Introductory Course?Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=18
3
3
4
2
6
This is our second view
How many participants at the course?
• Capture: Source ”View #1”• Recapture: Source ”View #2”
• Estimations
• Assumptions hold?
Number of participants
N = 33 * 18 / 13 = 47
Sensitivity of View # 1 Sn1 = 33/47 = 70.2%Sensitivity of View # 2 Sn2 = 18/47 = 38.3%
Yes No
Yes 13 20 View #1 = 33
No 5 x
View # 2 = 18N = 13 + 20 + 5 + x
Source View #2 – After Dinner Tutorial
Source View #1Dinner
How many persons are in the EPIET 2011 Introductory Course?Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=20
3
3
4
2
6
This is our second view (revisited)
+ 2
Number of participants
N = 33 * 20 / 13 = 51
Sensitivity of View # 1 Sn1 = 33/51 = 64.7%Sensitivity of View # 2 Sn2 = 20/51 = 39.2%
Yes No
Yes 13 20 View #1 = 33
No 7 x
View # 2 = 20N = 13 + 20 + 7 + x
Source View #2, revised – After Dinner Tutorial
Source View #1Dinner
So, just how many are there?
2
9 25
Isla del Lazareto, Katharina‘s Lecture, Monday, 11 October 2010 – Case definition: “Persons in room“, N=53
9
18
9
30
5 off screen
The problem with the X:Finding a comprehensive view
Assumptions may not hold
1. The population is closed - Usually possible
2. Individuals captured on both occasions can be matched - OK if good recording systems
3. For each sample, each individual has the same chance of being included - Rarely true
4. Capture in the second sample is independent of capture in the first - Rarely true
Sources are independent(most important condition)
Being in one source does not influence the probability of being in the other source
bc
ad OR
OR > 1 (positive dependence): underestimates N
OR < 1 (negative dependence): overestimates N
Yes No
Yes a b Z1
No c d
Y1 N
Source Y
Source Z
Dependent sources
• Estimation of number of IVDU in Bangkok in 1991 (Maestro 1994)
• Two sources used: Methadone programme (April – May 1991) Police arrests (June – September 1991)
• Methadone Need for drugs Probability of being arrested = negative dependence, overestimation of N
Usefulness of capture-recapture
• If conditions are met Great potential to estimate population size by
using incomplete sources Cheaper than exhaustive registers or full counting
• Two sources Impossible to quantify extent of dependence
• Multiple sources Can adjust for dependence and variable
catchability
Examples of capture-recapture
• STDs in The NL Reintjes et al. Epidemiol Infect 1999
• Foodborne outbreaks in France Gallay et al. Am J Epidemiol 2000
• Pertussis in England Crowcroft et al. Arch Dis Child 2002
• Invasive meningococcal disease Schrauder et al. Epidemiol Infect 2006
Representativeness A representative system accurately describes
Occurrence of a health event over time Distribution in the population by place and time
Difficult to determine Compare reported events with actual events Characteristics of the population Natural history of condition, medical practices Multiple data sources
Related to data quality, bias of data collection, completeness of reporting
Timeliness
Disease onset
Diseasediagnosed
Reporting
of event
Action taken
Analysis and interpretation
Clinician, labs Public Health Authorities
SensitivityRepresentativeness
Predictive value positive
TimelinessAcceptability
FlexibilitySimplicity
Cost
Buehler’s balance of attributes
• Recommendations of evaluation Continue Revise Stop
If revising Increase participation rate of sources Simplify notification Increase the frequency of feedback Broaden the net . . . Activate data collection
Improving surveillance systems
Surveillance is like archeology of the immediate past –It requires your responsible imagination of an invisible reality.
Carnunthum, Austria
Corollary
Thank you!
Literature
• CDC. Updated guidelines for evaluating public health
surveillance systems. MMWR 2001; 50 (RR-13): 1-35
• WHO. Protocol for the evaluation of epidemiological
surveillance systems. WHO/EMC/DIS/97.2.
• Romaguera RA, German RR, Klaucke DN. Evaluating
public health surveillance. In: Teutsch SM, Churchill RE,
eds. Principles and practice of public health surveillance,
2nd ed. New York: Oxford University Press, 2000.
Reading on capture-recapture
• Wittes JT, Colton T and Sidel VW. Capture-recapture models for assessing the completeness of case ascertainment using multiple information sources. J Chronic Dis 1974;27:25-36.
• Hook EB, Regal RR. Capture-recapture methods in epidemiology. Methods and limitations. Epidemiol Rev 1995; 17: 243-264
• International Working Group for Disease Monitoring and Forecasting. Capture-recapture and multiple-record systems estimation I: History and theoretical development. Am J Epidemiol 1995;142:1047-58
• International Working Group for Disease Monitoring and Forecasting. Capture-recapture and multiple-record systems estimation II: Applications in human diseases. Am J Epidemiol 1995;142:1059-68