exercise evaluation
DESCRIPTION
IAEA Training in Emergency Preparedness and Response. Exercise Evaluation. Module L-056. Lecture. Objectives. Understand the importance of evaluation for the success of the exercise Understand performance-based evaluation Know the role of evaluators and their procedures - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/1.jpg)
IAEA Training in Emergency Preparedness and Response
Exercise Evaluation
LectureLecture
Module L-056Module L-056
![Page 2: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/2.jpg)
L-056: Exercise Evaluation 2
Objectives
• Understand the importance of evaluation for the success of the exercise
• Understand performance-based evaluation
• Know the role of evaluators and their procedures
• Be able to develop evaluation criteria
• Know how to grade and report evaluations
![Page 3: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/3.jpg)
L-056: Exercise Evaluation 3
Principles of Evaluation
• Performance required by plans and procedures is being tested
• Evaluations are results-oriented to enhance the plan and the supporting elements, such as proper organization, training, and equipment
![Page 4: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/4.jpg)
L-056: Exercise Evaluation 4
Aim of Evaluations
• The aim of an evaluation is to identify strengths and weakness in the emergency response capability
![Page 5: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/5.jpg)
L-056: Exercise Evaluation 5
What is a Performance-based Evaluation?
• Focus on results, not on process
• Based on response objectives
• It answers the question: was the result achieved?
![Page 6: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/6.jpg)
L-056: Exercise Evaluation 6
How to Evaluate Based on Performance (1)
• Define the response or exercise objective• e.g. Protection of facility personnel
• Define the criteria• e.g. The personnel are promptly alerted
• e.g. The dose to personnel is kept within established limits
![Page 7: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/7.jpg)
L-056: Exercise Evaluation 7
How to Evaluate Based on Performance (2)
• For each performance objective, define the evaluation criteria• e.g. The personnel are promptly alerted
• Was a public address message broadcast immediately upon discovery of an emergency situation?
• Did all personnel hear the message?
• e.g. The dose to personnel is kept within established limits
• Were personnel relocated to safe areas?
• Was the dose rate monitored at personnel assembly areas?
![Page 8: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/8.jpg)
L-056: Exercise Evaluation 8
How to Evaluate Based on Performance (3)
• After the exercise, based on the FACTS, answer the question: “was the performance objective met”?
• It is not absolutely necessary that all criteria be satisfied• The actual response will depend on the situation
![Page 9: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/9.jpg)
L-056: Exercise Evaluation 9
How to Evaluate Based on Performance (4)
• If the performance objective was met• Was the procedure followed?
• Was the result achieved in good faith or by emergency?
• If the performance objective was not met, why?• Was the procedure followed?
• Was training lacking?
• Was equipment lacking?
• Was the exercise at fault?
![Page 10: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/10.jpg)
L-056: Exercise Evaluation 10
Evaluation Team
• Organization• Chief evaluator
• Evaluators for all key groups or locations
![Page 11: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/11.jpg)
L-056: Exercise Evaluation 11
Evaluator Role and Responsibilities
• Chief evaluator• Coordinate the evaluation team
• Instruct evaluators on the schedule, method and criteria
• Ensure consistency between evaluators
• Instruct the evaluators on the format and schedule of the evaluation reports
• Produce the final evaluation report
![Page 12: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/12.jpg)
L-056: Exercise Evaluation 12
Evaluator Role and Responsibilities
• Evaluators• Observe and record
• Do not evaluate during the exercise
• Stick to facts
• Evaluate after the exercise
• Provide a report to the chief evaluator
![Page 13: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/13.jpg)
L-056: Exercise Evaluation 13
Choice of Evaluators
• Evaluators must be qualified for the function they evaluate
• Evaluators must be trained• Evaluation techniques
• Procedures of the groups being evaluated
• Evaluators do not need to know the detailed scenario
![Page 14: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/14.jpg)
L-056: Exercise Evaluation 14
Preparation of the Evaluation Team
• Brief the evaluation team
• Provide clear written instructions
• Organize a site orientation tour
![Page 15: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/15.jpg)
L-056: Exercise Evaluation 15
Evaluator Instructions
• Before the exercise:• Review the scenario timeline and objectives
• Review the procedures used by the participants
• Review the applicable evaluation criteria
• Understand the actions required to demonstrate a performance objective
![Page 16: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/16.jpg)
L-056: Exercise Evaluation 16
Evaluator Instructions (Cont’d)
• During the exercise• Arrive before an event
• Locate a telephone
• Wear Evaluator’s identification
• Do not interfere with the participants
• Never criticize the participants
• Minimize communications
![Page 17: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/17.jpg)
L-056: Exercise Evaluation 17
Evaluator Instructions
• Following the exercise• Attend debriefing and critique
• Observe the effectiveness of the debriefing and critique
• Obtain exercise documents
• Ensure exercise location is in a safe state
• Finalize your chronology notes
![Page 18: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/18.jpg)
L-056: Exercise Evaluation 18
Observation Techniques
• Record arrival times
• Record names and functions
• Record times of major events
• Record repetitive interventions
• Listen to commands and instructions
• Listen and record Controller inputs
• Record exercise observations
![Page 19: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/19.jpg)
L-056: Exercise Evaluation 19
Guide for Evaluators
• After the exercise, based on the FACTS, answer the question: “was the performance objective met”
• It is not absolutely necessary that all criteria be satisfied:• The actual response will depend on the
situation
![Page 20: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/20.jpg)
L-056: Exercise Evaluation 20
Drillmanship
• Drillmanship is the art of exercise delivery
• Record facts on:• Simulation
• The exercise scenario
• Safety
• Participant feedback
![Page 21: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/21.jpg)
L-056: Exercise Evaluation 21
On the Use of Evaluation Checklists...
• Several organizations use checklists for each group being evaluated
• This focuses the evaluation on each group in isolation rather than on the overall coordinated response performance
• Focus on recording important facts rather than “checking” a list
• Evaluate the overall performance on the basis of exercise objectives using all the facts collected by the evaluators
![Page 22: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/22.jpg)
L-056: Exercise Evaluation 22
Debriefings
• Hot Wash-up
• Exercise Team Debriefing
• Executive Debriefing• Brief the management staff
• Give credit to the players for identifying deficiencies and give them a grace period to fix the problems before forcing compliance or imposing penalties
![Page 23: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/23.jpg)
L-056: Exercise Evaluation 23
Safety Responsibilities
• Evaluators are also responsible for monitoring exercise play to ensure a safe environment
![Page 24: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/24.jpg)
L-056: Exercise Evaluation 24
Grading Criteria
• Excellent
• Satisfactory
• Unsatisfactory
![Page 25: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/25.jpg)
L-056: Exercise Evaluation 25
Grading Criteria (cont’d)
• Grading must be supported by facts (if it was not observed, it didn’t happen)
• Trivial mistakes remain trivial even when they are observed
![Page 26: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/26.jpg)
L-056: Exercise Evaluation 26
Classification of Unsatisfactory Grades
• Critical Deficiency
• Major Deficiency
• Minor Deficiency
![Page 27: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/27.jpg)
L-056: Exercise Evaluation 27
Deficiency Classification
• Planning
• Training
• Resources
![Page 28: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/28.jpg)
L-056: Exercise Evaluation 28
Exercise Report
• A useful report is most difficult to write
• Based on facts, not impressions
• Focus on key areas
• Use the exercise objectives as the basis for the report
• Prepare and publish in a timely fashion
• Not everybody will agree with the findings
![Page 29: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/29.jpg)
L-056: Exercise Evaluation 29
Post Exercise Actions
• Report clarification• Ensure there are no inaccuracies in the report
• Critical deficiencies• Take corrective actions as soon as possible, e.g. one
month
• Major deficiencies• Take corrective actions within a reasonable time frame,
e.g. months
• Minor deficiencies• Take corrective actions during the annual revision of the
plan or procedures
![Page 30: Exercise Evaluation](https://reader030.vdocuments.us/reader030/viewer/2022033107/56814109550346895dacce8b/html5/thumbnails/30.jpg)
L-056: Exercise Evaluation 30
Summary
• The evaluation should be performance-based
• Evaluators and controllers are not the same persons
• An evaluator does NOT evaluate DURING an exercise, but after
• Base the evaluation report on the exercise objectives
• Look for corrective actions