![Page 1: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/1.jpg)
“To Ignore or Not to Ignore?”
Follow-up to Statistically Significant Signals"
Biosurveillance Information Exchange Working Group
Reflections from San Diego County
Jeffrey Johnson, MPHSan Diego County Health & Human Services Agency
2/23/06
![Page 2: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/2.jpg)
SAN DIEGO COUNTY
• Nearly 3 million population
• International border
• Large military presence
• Biotechnology Hub
• 21 Emergency Departments
![Page 3: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/3.jpg)
Early Event Detection in San Diego
• Evolving effort since pre - 9/11
• Data sources: ER Visits, Paramedic transports, 911 calls, school surveillance, OTC sales
• Systems: Local SAS/Minitab system, ESSENCE, and BioSense
• Statistical Methods: Descriptive, time series, CUSUM, EWMA, process
control methods (P&U Charts)
• Multiple syndromes • Visualization and alerting• Incident Characterization• Follow-up to signals County of San Diego
Health & Human Services Agency
![Page 4: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/4.jpg)
If We Ignore A Signal……
• We take no action or follow-up
• Save staff resources
• Avoid bothering hospital staff yet again
• Another data source may signal
• “The Feds may pick it up”
• Might lose an earlier start to a response
• We might be dead wrong to ignore
![Page 5: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/5.jpg)
If We Do Not Ignore a Signal……
• Will it be another “false alarm”
• May detect an event earlier
• Earlier response
• Continued interaction with the medical community
• Gain experience with follow-up
• Increased situational awareness
![Page 6: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/6.jpg)
Characterization of Detections
• Detection Method • Syndrome group• % Admitted• Deaths?• Geographic cluster?• Prior day’s level?• Recent level?• Age groups?• Severe syndrome?• Detections in other data sources?• Other epidemiological intelligence?• Other diagnostic information
Follow-up?
Action or
No Actionor
Watch
![Page 7: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/7.jpg)
Detection Follow-up with Medical Community
What is the final diagnosis of Patients A, B, C?
Is there a common pattern among admitted patients?
Did any have lab test results that might suggest a larger event?
Among patients with a common zip code, was there a shared living setting or common exposure?
Can we send someone out to review medical charts?
What is your facility’s assessment of the situation?
County of San DiegoHealth & Human Services Agency
![Page 8: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/8.jpg)
County of San DiegoHealth & Human Services Agency
Routine Surveillance Activities
Rule out system error
YES
NO
Preliminary evaluation
Describe initial results
True Positive
Aberrationdetected
Potential false positive
YES NO“False Positive”
Inform key divisional staff
Intensivemonitoring & surveillance
Evaluate otherdata sources
Cluster check
VERIFY
NOTIFYInform key departmental staff
IDENTIFY
Ignore?
Ignore?
Ignore?
Ignore?
![Page 9: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/9.jpg)
0
10
20
30
40
50
60
70
80
90
10/31/2004 11/30/2004 12/31/2004 1/31/2005 2/28/2005 3/31/2005 4/30/2005 5/31/2005 6/30/2005 7/31/2005
Gastrointestinal 7 Day Moving Average
0
5
10
15
20
25
30
10/31/2004 11/30/2004 12/31/2004 1/31/2005 2/28/2005 3/31/2005 4/30/2005 5/31/2005 6/30/2005 7/31/2005
Gastrointestinal 7 Day Moving Average
0
5
10
15
20
25
10/31/2004 11/30/2004 12/31/2004 1/31/2005 2/28/2005 3/31/2005 4/30/2005 5/31/2005 6/30/2005 7/31/2005
Gastrointestinal/Genitourinary 7 Day Moving Average
GI Syndrome Over Time (10/31/04 – 8/24/05)
ED
911
Par
amed
ic
Ru
ns
![Page 10: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/10.jpg)
The Significant Aspects of Syndromic Surveillance
• Statistical Significance
• Public Health Significance
• Significant Event
• Significant Public Awareness
• Significant Biological Agent Detection
![Page 11: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/11.jpg)
HAZMAT FLAG – 12/04/2004
County of San DiegoHealth & Human Services Agency
Statistical significance vs. public health significance
![Page 12: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/12.jpg)
County of San DiegoHealth & Human Services Agency
Statistical significance vs. public health significance
![Page 13: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/13.jpg)
County of San DiegoHealth & Human Services Agency
![Page 14: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/14.jpg)
Syndromic Surveillance for Natural Disasters
San Diego Wild Fires, 2003
Significant event with statistically significance outcomes
San Diego County
![Page 15: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/15.jpg)
Syndromic surveillance for natural disasters
Significant event with statistically significance outcomes
![Page 16: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/16.jpg)
San Diego County: Normalized Prehospital Transports and ED Visits with "Chest Pain" as Chief Complaint (12/31/03 - 10/08/04)
-3
-2
-1
0
1
2
3
4
5
12/31/03 1/31/04 2/29/04 3/31/04 4/30/04 5/31/04 6/30/04 7/31/04 8/31/04 9/30/04
No
rmal
ized
Co
un
t
ED Visits Ambulance Runs
San Diego County: Prehospital Transports and ED Visits with "Chest Pain" as Chief Complaint (12/31/03 - 10/08/04)
0
10
20
30
40
50
60
12/31/03 1/31/04 2/29/04 3/31/04 4/30/04 5/31/04 6/30/04 7/31/04 8/31/04 9/30/04
Co
un
t
ED Visits Ambulance Runs
“The Clinton Effect”
September 4, 2004
While spikes in both datasets are apparent, normalized counts show a relatively larger increase in ED visits on Sept. 6, 2004.
Significant Public Awareness
![Page 17: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/17.jpg)
7/7/05 London Bombings
San Diego CountyParamedic Transports
for “Chest Pain”
Significant Public Awareness
![Page 18: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/18.jpg)
Application of Syndromic Surveillance Agent:
Syndrome categoriesSpecific word search in CC or DX fields
Sensor site:Zip codes, population (schools)
Date:Temporal based surveillanceNew pre-detection baselines
Biowatch
BioWatch Detection• Tells us agent, sensor site and date• Plume plot may help us narrow surveillance on a geographic area
Significant BT Agent Detection
![Page 19: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/19.jpg)
Anatomy of a Detection(a case example)
![Page 20: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/20.jpg)
Daily Email Report
Attached Table
Feb 5, 2006 911 Call Data
![Page 21: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/21.jpg)
911 Call Center - GI Syndrome Signal
![Page 22: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/22.jpg)
![Page 23: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/23.jpg)
Line listing for review
Non-specific call
complaints
![Page 24: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/24.jpg)
21 Signals since 07/01/03
Various statistical signals
The count for the signals include a consistent range
911 Call Center - GI Syndrome Signal
![Page 25: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/25.jpg)
What did we do?
• Magnitude of cases
• Which method(s) signaled?
• Check the other call centers
• Check the other data sources
(ED data, EMS transports)
• Review the line listing
• Our conclusion…..
…... 14 vs mean of 7.8
…... CUSUM (2), P-Chart, U-Chart
…… No signals
…… No Signals
……. No apparent pattern
>>>>> • Super Bowl Sunday • Fewer trauma calls • Smaller denominator (P-Chart)• Traditional increase in GI on
this day• Watch next day’s results
![Page 26: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/26.jpg)
Case Example #2
Hospital 9 ED DataRespiratory Syndrome
![Page 27: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/27.jpg)
Hospital 9 - Daily Results Table
![Page 28: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/28.jpg)
![Page 29: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/29.jpg)
![Page 30: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/30.jpg)
0
5
10
15
20
25
30
35
40
1/1/2004
2/1/2004
3/1/2004
4/1/2004
5/1/2004
6/1/2004
7/1/2004
8/1/2004
9/1/2004
10/1/ 2004
11/1/2004
12/1/ 2004
1/1/2005
2/1/2005
3/1/2005
4/1/2005
5/1/2005
6/1/2005
7/1/2005
8/1/2005
9/1/2005
10/1/ 2005
11/1/2005
12/1/ 2005
1/1/2006
2/1/2006
Count Signal
Hospital 9 Respiratory Syndrome
01/01/04 - 02/03/06
Many signals….
So what’s the context?
Do we ever ignore the signals?
![Page 31: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/31.jpg)
Hospital 9 Respiratory Syndrome
• 24 signals over a 37 day period• Count range: 11 – 34• Over time an increasing mean
![Page 32: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/32.jpg)
Hospital 9 Influenza-like-illness (ILI) Syndrome
• “ILI syndrome” has greater syndrome specificity than “Respiratory” syndrome”
• 16 signals over a 37 day period
Greater Syndrome Specificity……
![Page 33: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/33.jpg)
• S$gn&ls Happen!
• Make sure you see flames before yelling “Fire”
• CUSUM 2 & 3 STD may be too sensitive
• We lose precision with non-specific syndromes
• Everyone wants to know what’s going on all the time
• Increasing focus on situational awareness
• Further evaluation and testing required
What We Have Learned
![Page 34: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/34.jpg)
Event or
Technology Trigger
Peak of
Inflated Expectations
Trough of
Disillusionment
Slope of
Enlightenment
Plateau of
Productivity
Hype Cycle of Emerging “Syndromic Surveillance” Technologies
Adapted from the Gartner Hype Cycle
Too many signals?, IT Costs, poor syndrome specificity, evaluation
results
Prioritized data sets, protocols in place,
The “magic bullet”
9/11, Anthrax attacks
Dual use, situational awareness, appropriate
signals
![Page 35: “To Ignore or Not to Ignore?” Follow-up to Statistically Significant Signals" Biosurveillance Information Exchange Working Group Reflections from San Diego](https://reader035.vdocuments.us/reader035/viewer/2022062407/56649d805503460f94a64631/html5/thumbnails/35.jpg)
• More work in all areas of syndromic surveillance is needed
• Knowledge requires responsibility
• The enemy is studying our efforts
• Current/future funding levels require reliability, efficiency and sustainability of systems and approaches
• The Future:Neural networks and Artificial Intelligence (AI)?
• Are we ready?
Considerations