exercise design, conduct and evaluation · exercise design, conduct and evaluation overview for...
TRANSCRIPT
Exercise Design, Conduct and Evaluation Overview for ACFs
Adult Care Emergency Preparedness Webinar Series
April 28, 2015
Michael McCollum
Director of Exercises & Training
New York City Department of Health and Mental Hygiene
Office of Emergency Preparedness and Response
Why Exercise?
Exercise Design and Development
Exercise Conduct
Exercise Evaluation
Q&A
Adjourn
Agenda
This is just an overview
Not an “official” HSEEP certified course
Further resources will be provided later
Clarification
DOHMH History
Over 68 exercises in 12 years
These include:
City Hall BioWatch TTX, 2003 NYC SNS Drill (10 PODs), 2004 PODEX FSE (4 PODs), 2005 Emergency Communication WS, 2006 REOP TTX, 2007 Coastal Storm TTX, 2008 ICS Leadership White Powder TTX, 2009 PIO COT Call Down Drill, 2010 MH Essential Services WS, 2011 PIB Staging Drill, 2012 Multiagency Environmental Sampling Drill, 2013 Rapid Activation of Mass Prophylaxis Exercise (RAMPEx), 2014
Why Exercise?
A simulation designed to improve preparedness and emergency operations
A response based on mock events to motivate realistic actions
Players react as they would in real life
What is an Exercise?
Why Exercise?
Staff Experience
Recommendation
Test Plans
Line in the Sand
Push emergency planning
Staff plan review
Preparedness Cycle
Exercise Type
Discussion Based
Seminars/Orientations
Workshops
Tabletop Exercises (TTX)
Operations Based
Drill
Functional Exercise (FE)
Full Scale Exercise (FSE)
Building Block Approach
Discussion-based Exercises
Familiarize participants with current plans, policies and procedures or with facilitating the development or revision of plans
• Seminars – Orient participants to new or updated plans, policies or procedures in an informal discussion
• Workshops – Build plans or policies or test a new plan using a canned scenario
• Tabletops (TTX) – Assess plans, policies and procedures in scenarios
involving key stakeholders
Operations-based Exercises
Validate plans, policies and procedures, clarify roles and responsibilities and identify resource gaps in an operational environment
• Drills – Test a specific operation/function of a single entity or team
• Functional Exercises (FE) – Validate communications and command and
control functions among multi-agency coordination centers (a.k.a. Command Post Exercises)
• Full-Scale Exercises (FSE) – Evaluate coordinated operations among multiple agencies, jurisdictions and disciplines with "boots on the ground" in real time, real place scenarios
Differences in Exercise
Discussion
Slide Development & Facilitation
Situation Manual (SitMan)
Conference & Breakout
Scribing & Evaluation
Relatively Quick/Cheap
Operations
Exercise Plan (ExPlan)
Master Scenario Events List (MSEL)
Multiple Sites
Evaluators & Exercise Evaluation Guide (EEG)
Safety Concerns
Lengthy Process/Expensive
Workshops, tabletops (TTX) and drills can be low cost and developed in-house
Full-scale exercises are typically the most expensive and require the most external support
Differences in Exercise
Exercise Design & Development
Design & Development
Design
Assessing exercise needs
Defining the exercise scope
Writing a purpose statement
Defining exercise objectives
Creating an exercise scenario
Development
Creating documentation
Arranging logistics, actors, and safety
Coordinating participants
Supporting tasks (e.g., training controllers, evaluators, and exercise staff)
Planning Team
Purpose
• Determines exercise objectives
• Tailors scenarios to meet objectives
• Determines logistical support
Composition
• Exercise Director
• Subject Matter Expert
• Planning Coordinator
• Lead Evaluator
• Section Liaison
Model on Incident Command System
• Logistics
• Safety
• Finance
Planning Schedule
Meeting Tabletop Functional Full Scale
Initial Planning 3 Months Prior 6 Months Prior 12 Months Prior
Midterm Planning 4-6 Weeks Prior 3 Months Prior 6 Months Prior
Master Scenario Events List Meeting
NA 2 Months Prior 3 Months Prior
Final Planning 2-3 Weeks Prior 1 Month Prior 1 Month Prior
CONDUCT
Draft After Action Report 45 Days After
Final After Action Report 60 Days After
Situation Manual (SitMan)
Slide Set
Facilitator Manual
Documentation – Discussion Based
Exercise Plan (ExPlan)
Exercise Evaluation Guide (EEG)
Ground Truth Documents
Mock Client Data
DAL
Documentation – Operations Based
Objectives
Exercise objectives are the distinct outcomes that an organization wishes to achieve during an exercise.
Keeps the exercise in scope and avoid mission creep SMART objectives facilitate effective scenario design,
exercise conduct, and evaluation
Objective: Utilize DAL 14-15: Warm Weather Advisory to draft facility management plan to protect clients within an hour S: Provides the 5 Ws M: Simple deadline, unit of time A: Achievable, depending on group conducting exercise R: Planning is vital to emergency response T: Unit of time gives definitive way to measure if objective is
achieved or not
Scenario: the National Weather Service (NWS) has forecasted a heat index of 100°F for the next 3 days
Objectives Example – Tabletop (TTX)
Objective: Execute eFINDS plan to facilitate evacuation of 100 residents in one hour S: Provides the 5 Ws
M: Simple deadline, unit of time
A: Achievable, depending on group conducting exercise
R: Communications are vital to emergency response
T: Unit of time gives definitive way to measure if objective is achieved or not
Scenario: Category 5 hurricane scheduled to make landfall across multiple jurisdictions within next 48 hours
Objectives Example - Drill
Provides the simulated backdrop that both guides and drives all exercise activities
Provides realism to put players in the right mindset
Should stress but not overwhelm
Should be crafted after objectives are made to ensure they will be addressed
Scenario
Where does the initiating event take place?
What is the impact of the incident?
What time of day does the event take place?
What is the sequence of events?
What other factors would influence emergency procedures?
Scenario
Who needs to participate in order to accomplish scenario objectives?
How do we foster participation? Do you have buy in from
leadership?
How far in advance should they be invited?
How many participants can be included?
How many observers should we expect?
Participant Considerations
Players
Player Level
Leadership
Management
Support Staff
Field Personnel
Sample Activity
Assess Policy & Procedure
Test Command & Control
Show Communications Ability
Demonstrate Hands-On Skills
Exercise Conduct
Tabletop vs. Operations
Tabletop (TTX)
Facilitator
Slide Sets
Situation Manual
Conference Room
Break Outs
Scribes
Operations
Exercise Briefings
Master Scenario Events List (MSEL)
ExPlan & C/E Plan
Exercise Evaluation Guide (EEG)
Multiple Sites
Communication needs
8:00 a.m. Registration
8:30 a.m. Welcome and Introduction
8:45 a.m. Module 1 - Pre-incident/Warning
9:45 a.m. Module 2 - Notification and Initial Response
10:45 a.m. Module 3 - Response
11:30 a.m. Review and Conclusion
Example TTX Schedule
Facilitators: ensure focused discussions
Controllers: manage operations-based exercises
Players: those for whom the exercise is conducted
Actors: simulate specific roles (i.e., facility residents) to add realism
Evaluators: records discussions and activities to inform after action report
Observers: must not participate in exercise play
Exercise Staffing
Exercise Staffing
Participant Type Badge Color
Player Green
Evaluator Red
Controller Teal
Safety Yellow
Start Exercise (StartEx) should be at a designated time
End Exercise (EndEx) occurs when
All exercise objectives have been met
Or allocated time has been used
Determined by Lead Controller
StartEx/EndEx
Exercise Evaluation
Methodology
Discussion based
Plans, policies and procedures
Resources and capabilities
Interagency/inter-jurisdictional relationships
Operations based
Effectiveness of communications
Tasks performed correctly
Ability to properly use equipment
Exercise Evaluation Guide
Player Debriefing
As close to EndEx as possible
Immediate impressions
What went well, what needs to be improved
Lead by a facilitator and at least two scribes
Hotwashes
Used to gain feedback from everyone involved
Paper-based
Paper-based need to be distributed onsite
Participants can’t leave until done
Determines if participants felt objectives were met
Include some open-ended questions to gain concerns
Participant Surveys
Should be prepared after every exercise type
Summarize what happened during the exercise
Provide feedback to participants
Recommend improvements
After Action Reports
Executive Summary
Exercise Overview
Exercise Goals and Objectives
Exercise Events Synopsis
Analysis of Mission Outcomes
Analysis of Critical Task Performance
Conclusion
Appendix: Improvement Plan Matrix
Recommended Format for After Action Reports (AAR)
Improvement Plan
Converts recommendations from the After Action Report (AAR) into measurable steps
States who will be responsible for those steps
Provides a timeline for implementation
Can inform future activity
HSEEP Toolkit (document templates): https://hseep.preptoolkit.org/HSEEP.html
FEMA Independent Study: https://training.fema.gov/is/
IS-120.a An Introduction to Exercises
Public Health Emergency Toolkit: http://www.cidrap.umn.edu/sites/default/files/public/php/339/
339_toolkit.pdf
Further Resources
Questions?
