facilitator/evaluator orientation/briefing marc mass fatality ttx may 21, 2013 1
TRANSCRIPT
2
• Roles and Responsibilities• Evaluation Methodology Overview
– Exercise Evaluation Guide (EEG) Concept
• Evaluation Process– Evaluator Responsibilities
• Assessment, Review, and Analysis of the Exercise
Briefing Topics
Roles and Responsibilities
• Facilitators provide situation updates and moderate discussions. They also provide additional information or resolve questions as required.
• Evaluators use EEGs to measure and assess performance, capture unresolved issues, and analyze exercise results. Evaluators passively assess and document players’ performance against established emergency plans and exercise evaluation criteria, in accordance with HSEEP standards. Evaluators have a passive role in the exercise and only note the actions/decisions of players without interfering with exercise flow. For this exercise, evaluators may provide a limited coaching role of stimulating thought process.
4
Facilitators Evaluators
• Chris Kates – UC• John Linstrom- MF• Justin Sorg – EM/Logs• Joan Harding– Health
and Medical• Dwight Graves PIO • Dana Hall VIC
• UC – Joe Lynch • MF – Chuck Smith• EM Logs – Ricky
Carroll• Health & Medical – Erin
Lynch• PIO Dwight Graves• VIC – Lisa Ladue
5
• Validate strengths, and identify improvement opportunities
• Validate plans, procedures, and protocols
• Improve processes and plans• Determine level of capability in regard to
the exercise’s core capabilities
Evaluation Goals
• The Homeland Security Exercise and Evaluation Program (HSEEP) Volume III provides evaluation and improvement planning policy.
• Evaluations should be objective-based to remain in line with:
–National Preparedness Goal–Core Capabilities List–Jurisdiction and agency plans
• HSEEP EEG templates provide streamlined process for standardized evaluation concept, not necessarily content.
EEG Concept
6
• Allows evaluators to review validation of plan elements– Plans used will depend on participating agencies
• Includes references to plan sections, tables, and charts– Allows evaluators to better prepare for exercise– Provides detail for what will be expected of players
• Remains focused on exercise objectives as they relate to the selected core capability
EEG Function
7
• Evaluation team should use EEGs in conjunction with full plans to prepare
• Use EEGs as guides for observation, but do not focus on completing them during the exercise
Use EEGs for data analysis after exercise and in preparation for the evaluator debriefing
Use measured results to make actionable improvement recommendations within the After Action Report (AAR)/ Improvement Plan (IP)
Use of EEGs
8
HSEEP Methodology Steps
Evaluation Tools
Timeline
Exercise objectives,
actual plans and
procedures
Evaluator log EEG analysis sheet
AAR data input form
Through Final Planning
Conference (FPC) planning
During exercise
Controller/ Evaluator
(C/E) Debriefing
Post exercise, before AAR
draft development
Use of Evaluation Tools
9
Evaluator Notes and Data
• Evaluators should capture player discussions, decisions, actions, and activity during exercise play. Use back of EEGs for notes
• The EEGs provided should serve as guides for documentation of exercise activity
10
EEG Analysis Form
• Each evaluator will be provided with EEGs that will give specific guidance on what data to collect during the exercise, how to record it, and how to analyze it
• These forms should be completed at the conclusion of the exercise, using data gathered during the exercise
11
EEG AAR Data Input Form
• Each EEG will include a second form to be completed following the C/E Debriefing
• This form will serve as an expansion to the notes captured in the EEG analysis form, including an area to provide recommendations.
• Evaluators should complete one AAR input form for each identified strength or area for improvement
12
After Action Report (AAR)/Improvement Plan (IP)
• Data from the EEGs and debriefing will be compiled into the AAR/IP.
IP
13
• Be on time• Wear appropriate attire• Make sure you have the tools necessary
to perform task• Understand the exercise concept and
scenario • Read the sit man
Facilitator Evaluator Responsibilities
14
• Be familiar with the assigned EEGs• Collect information• Make observations on player
discussions, knowledge, and intended impact of decisions
• Analyze actions and develop post exercise AAR/IP input
Evaluator Responsibilities
15
• Review appropriate plans, procedures, and protocols
• Participate in C/E Briefing• Review appropriate exercise materials to
be familiar with scenario, objectives, and core capabilities
• Review EEGs and other supporting materials for your area of responsibility
Pre-exercise Responsibilities
16
• Monitor and document player performance• Use EEGs to confirm objectives are being
met• Focus on critical tasks, as specified in the
EEGs and plans being exercised• Do not prompt players or interfere with
exercise play• Limited coaching is allowed
Exercise Responsibilities
17
At the start of the exercise, evaluators should:• Position themselves to appropriately hear
discussions• Be prepared to move between break out groups
if necessary to follow various discussions
Evaluators should not:• Focus solely on individuals• Interfere with exercise discussion or action
Evaluator Guidelines
18
Evaluators must keep an accurate written record of important issues that take place during the exercise: Were participants knowledgeable of plans and
procedures to meet exercise objectives? Did discussions validate that operations would occur
in accordance with plans and procedures? Did participants validate that sufficient personnel are
equipped, trained, and assigned to perform tasks?
Observing the Exercise
19
• Evaluators should record events such as:–Discussed actions of players–Key decisions–Discussions that denote deviations from plans and procedures
–Player discussion of key movements and activities–Module responses on easel pad–Hotwash responses
Recording Important Events
20
Postexercise Analysis Responsibilities
• Participate in the Hot Wash, and take notes on findings identified by players
• Summarize your notes and prepare for the C/E Debriefing• Have your summary ready for the lead evaluator, including
bulleted strengths and areas for improvement from activity analysis
• Use data to determine whether the exercise capabilities and plans were effectively implemented– Identify observations as strengths or areas for
improvement
21
• Allows players to participate in a self-assessment of the exercise play
• Allows evaluators to clarify additional points and collect missing information
• Identify three sustains and three improves• Make sure sustains and improves are
listed on the easel pad
Player Hot Wash
22
• Work to complete EEG analysis forms for submission
• Discuss degree to which each objective was completed
• Provide strengths, best practices, lessons learned, and possible IP recommendations
• Begin outlining issues to be included in AAR/IP on AAR data input forms
Controller and Evaluator (C/E) Debriefing
23
AAR/IP Input
• Use the EEG AAR data input form• Complete one form for each strength and area for improvement.
– Observation. Write a one-sentence statement that identifies the strength or area for improvement as noted in the activity analysis tables.
– References. Reference relevant plans, policies, procedures, laws, or regulations that could be used to facilitate implementing recommendations for improvement or capturing strengths.
– Analysis. Document detailed data on what you observed during the exercise specific to this individual observation.
– Recommendations. Suggest steps to allow the participants to further improve the plans or share strengths.
• All EEG input is due to Joe or Lou no later than departure from the exercise site on the exercise day.
24
Evaluation Matrix
Objective Met Objective Not Met
Players followed plan procedures
Plan works
Document best practices and strengths
Capture why plan didn’t work
Document how to improve relevant plan elements
Players did not follow plan procedures
Capture how objective was met
Recommend successful actions and procedures be included in plan rewrite
Capture why plan was not used
May require plan training and familiarization
25
Evaluator Deliverables
• Before you leave–Completed EEGs–Notes–Pictures of easel pads in your
section (Joe has camera and will do this but easel sheets are important to retain)
26