evaluation of training and education activities
DESCRIPTION
Evaluation of Training and Education Activities. Objectives. By the end of this presentation, participants will be able to List reasons why evaluation is needed Describe different types of evaluation Discuss how to use different types of evaluation for training and education activities. - PowerPoint PPT PresentationTRANSCRIPT
Evaluation of Training and Education Activities
2
ObjectivesBy the end of this presentation, participants will be able to
List reasons why evaluation is needed Describe different types of evaluation Discuss how to use different types of
evaluation for training and education activities
Evaluation (1)
As trainers, what do we want to evaluate?
3
Evaluation (2) What have we accomplished?
What was the training? Where? How many people were trained? Cost?
How well have we done? Did the trainings go as planned? Did the participants like the training?
Was the training relevant? What were the strengths and
weaknesses?4
Evaluation (3) How effective is the training?
To what extent did the training achieve the objectives?
Did the participants increase their knowledge and skills?
Did they change their job performance? Did health status improve?
How can we improve on it? Are the training objectives appropriate,
given the available resources?5
Systematic Health Education Process
6
Development
Pilot-testing(Formative Evaluation)
Implementation(Process Evaluation)
Assess Effectiveness(Outcome and
Impact Evaluation) Evaluation
(Formative Research)
NeedsAssessment
7
When to Begin Evaluation?Consider evaluation up front (in planning phase of training project)
Allocate time Allocate resources Evaluation actually occurs at every
stage of development Determine the type of evaluation you
need
8
Levels of Evaluation Formative (before)
Identifies strengths and weaknesses of training/product before implementation
Process (during) Describes the implementation of the
training or distribution/use of the product
Summative/Outcome/Impact (after) Assesses the short-term and long-term
effects of the training/product
Four Types of Evaluation
9
Impact
Outcome
Process
Formative
Programimplementation
ends here.
Programimplementation
occurs here.
SummativeEvaluation
10
Goals and Objectives Goal
Describes the overall mission/purpose of the program Broad general statement that clearly states the
problem that will be addressed
Objectives State the expected result Specific, measurable, attainable, realistic, time-
framed Partial accomplishment of a goal
Activities Specific actions or procedures expected to occur to
achieve objectives
Formative Evaluation
11
Impact
Outcome
Process
Formative
Programimplementation
ends here.
Programimplementation
occurs here.
SummativeEvaluation
12
Formative Evaluation(Pilot Testing)
Conducted with the target audience Answers questions about whether
trainings/products are Understandable, relevant, attractive,
credible, acceptable, effective Identifies strong and weak points
Identifies confusing terms or concepts Allows for necessary revision prior to
implementation Prevents production of ineffective
products
Formative Evaluation Methods Focus groups
With various types of participants (e.g., experienced, non-experienced)
Interviews (key informant/audience) Informal discussion Open-ended questions
Understanding Appeal Utility
Test the curriculum in the “real” environment
Readability testing (literacy and comprehension) 13
14
Formative Evaluation Measures
Ease of use or intuitiveness Readability
Reading level Tone Fonts, colors, spacing
Comprehension Terminology and illustrations Concepts
TB:The Hollywood
Movie
Formative Evaluation Screen tests Movie screenings Focus groups
TB
Process Evaluation
17
Impact
Outcome
Process
Formative
Programimplementation
ends here.
Programimplementation
occurs here.
SummativeEvaluation
18
Process Evaluation(Monitoring of Training/Curriculum)
Examines the procedures and tasks involved in implementing a training/product
Assesses if the training/product is working as planned
Answers questions such as: Are the activities being implemented as
planned? Is the material being used as intended? How many participants attended the program? Were the participants satisfied with the
educational materials?
19
Process Evaluation Measures Product inventory
Hard copies distributed Web pages viewed or downloaded
Attendance Checklist and timetable review Budget, $$$$ spent Audience feedback on quality of
program/material during implementation (participant reactions)
Participant Reactions Do audience/participants think that:
Objectives are relevant to participants' work
Course content is realistic Training methods are suitable Facilitators are effective Training environment (classrooms, meals,
accommodation) contributes to learning
20
Participant Reactions Methods
Written feedback – participants write down their reactions Ensures that all participants provide comments,
not just the outspoken ones Both open- and closed-ended questions are asked
Oral feedback and observations Observations by facilitators Facilitators meet at the end of each day to
discuss their opinions and participant comments
21
Participant Reaction Form
22
Process Evaluation Ticket sales Critical reviews Rotten tomatoes
TB
Outcome Evaluation
24
Impact
Outcome
Process
Formative
Programimplementation
ends here.
Programimplementation
occurs here.
SummativeEvaluation
25
Outcome and Impact Evaluation
Answers the questions… What have you accomplished? How well have you done it? How much have you done? How effective have you been?
26
Outcome Evaluation Based on learning objectives Evaluation of learning & performance of
skills Assesses immediate or short-term effects
End of course or immediately after distribution Helps draw conclusions about effectiveness Answers questions such as:
Were learning objectives achieved? What knowledge and skills were gained as
a result of the training/product?
Outcome EvaluationEffectiveness of Self-Study Modules
on Tuberculosis Knowledge Mean Scores
Pre-test Post-testPrint 74% 91%Satellite 81% 92%Web 76% 92%
(p<.01)
27
Impact Evaluation
28
Impact
Outcome
Process
Formative
Programimplementation
ends here.
Programimplementation
occurs here.
SummativeEvaluation
29
Impact Evaluation Gathers information about the long-range
results of the training/product and improvements in health status 3 months, 6 months, one year following
implementation Answers questions such as:
Did the user’s job performance improve? What are the changes in job practice of user or
participant? What changes were implemented in the program?
Costly, time consuming, but important
30
Should You Measure Impact?
Ask yourself the following questions: Do you have the time and resources? Do you have support from management
and partners? Are you able to reach the audience at this
later date? What will motivate the audience to
participate?
Impact Evaluation Methods
31
Methods Surveys Key-informant interviews Direct observation/performance checklists Document/record reviews/treatment registers Reviewing of program indicators
Timeline Depends on the goal of the training;
however, generally 1 to 6 months after training
Summative Evaluation Domestic and international
box office revenues DVD sales Franchise opportunities
TB
Evaluation Reports
Evaluation Reports Justify program, course, or material to
stakeholders Allow for identification of areas for
improvement Provide evidence for additional needs Share lessons learned
Most Importantly…
Plan for evaluation at the beginning and conduct
throughout development process