class 13 chapter 9 evaluation of training
TRANSCRIPT
Class 13
Chapter 9
Evaluation of Training
Evaluation Phase
Evaluation
Objectives
Organizational
Constraints
Design
Issues
Evaluation
Strategy and
Design
Process
Measures
Outcome
Measures
• Reaction
• Learning
• Behavior
• Results
Input Process Output
Resistance to Training Evaluation
Argument Counterargument
There is nothing to evaluate. Evaluation is simply measuring
the degree to which training
objectives are achieved.
No really cares. Organization decision-makers
utilize training outcome data in
budgeting and staffing.
Evaluation is a threat to my
job.
Managers use evaluation as a tool
for improving the likelihood of a
successful intervention, not as a
pass/fail grade.
Evaluation = 2 Types
Process Evaluation
Outcome Evaluation
Purpose of Process Evaluation
Identify areas where training process
can be improved
Improved process leads to improved
outcomes
Types of Questions to Answer in a Process
Analysis (Before Implementation)
Were needs diagnosed correctly?
Were needs translated into training objectives?
Was an evaluation system designed to measure
objectives?
Was the training program designed to meet all the training
objectives?
Are the training methods appropriate for the learning
objectives?
Were the guidelines developed in the design phase used
to develop the training content, materials and processes?
Types of Questions to answer in a Process
Analysis (During Implementation)
Was the trainer and techniques well matched to the
objectives?
Were lecture portions of the training effective?
Was involvement encouraged/solicited?
Were questions used effectively?
Did the trainer conduct the various training methodologies
(case study, role-play, etc.) appropriately?
Was enough time allotted and was it used as intended?
Did trainees follow instructions?
Was there effective debriefing following exercises?
Was time allowed for questions?
Who Is Interested in the Process Data
Training
Department
Trainer: Yes, to determine what works well and what
does not.
Other trainers: Yes, to the extent that process is generalizable
Training manager Only if training is not successful or a problem
is shown for a particular trainer.
Customers
Trainees: No
Trainees’ supervisor No
Upper management: No
Outcome Evaluation
What do you evaluate at the
end of training???
Your Training Objectives
Outcome Evaluation
The KEY outcomes achieved by your training
Trainee Reactions
Learning
Transfer to the job
Organizational Results
The same as your objectives
Outcome Evaluation Design
Alignment Between TNA and Outcomes
Training Evaluation
Objectives Outcomes
Measured the
same way
TNA
Design EvaluationDetermines
Develop
Training
Objectives
Evaluation
Plan and
Design
Design Phase Activities Evaluation
Phase
Reactions
Learning
Performance
• Person
• Unit
• Org.
Reaction Questionnaires
Training reaction questionnaires do
not assess learning,
but DO assess the trainee’s
attitudes about and perception of
the training.
Reaction Questionnaires, continued
At a minimum, the following categories
should be assessed in a reaction questionnaire:
Relevance/value of training (to trainees)
Training content and materials
Trainer’s behavior/effectiveness
Facilities, activities and procedures
Reaction Questionnaires, continued
It should also provide an
overall evaluation of the training experience.
Steps to Consider in Developing a
Reaction Questionnaire – Part 1 of 2
1. Determine what you want to find out (consider training
objectives).
2. Develop a written set of questions to obtain the information.
3. Develop a scale to quantify respondents’ data.
4. Make forms anonymous so participants will feel free to respond
honestly.
5. Ask for subgroup information (e.g., young vs. old; minority vs.
non-minority).
• This could be valuable in determining effectiveness of
training for different cultures, for example, which may be
lost in an overall assessment.
Steps to Consider in Developing a
Reaction Questionnaire – Part 2 of 2
6. Allow space for “Additional Comments” in order to allow
participants the opportunity to mention things you might not have
considered.
7. Decide the best time to give the questionnaire to get the
information you want.
a. If right after training, ask someone other than the instructor
to administer and pick up the information.
b. If some time later, develop a mechanism to obtain a high
response rate (e.g., encourage the supervisor to allow
trainees to complete the questionnaire on company time).
Types of Outcomes and Examples of
Factors Affecting those Outcomes
Trainee characteristics
Trainer behaviorReactions
Trainee readiness for the course
Trainee motivation to learn
Design, materials, and content
Learning
Training’s transfer of training effectiveness
Motivational forces in the job setting
Opportunity to apply learned KSAs on the job
Job Behavior
External environment of the organization
Internal environment of the organization
Employee performance, KSAs, and needs
Organizational
Results
Examples of Assessing Objectives
Training objectives are set in the Design
Phase based on “Gaps” identified.
Training objectives are assessed in the
Evaluation Phase
Let’s look at some examples of how this is done for
reaction objectives.
Reaction Objectives - Example
At least 75% of participants will rate the following aspects of training as 4 or higher on a 5 point scale
(with 5 being the highest rating):
1. Quality of the training content
2. Relevance of the training content
3. Effectiveness of each trainer
4. Overall effectiveness
Training Content
Strongly Disagree - Strongly Agree
1 2 3 4 5Rating of
4 or 5Ave %
Quality 0% 0% 13% 40% 33% 73%
78%Relevance 0% 0% 13% 53% 33% 86%
Overall
Rating0% 0% 27% 47% 27% 74%
Table of Reaction Results
Ratings 1 = Low 5 = High
Trainer 1 2 3 4 5Rating of
4 or 5
Average
Rating
A 0% 0% 27% 41% 32% 73% 4.05
B 0% 13% 19% 36% 32% 68% 3.87
C 0% 0% 0% 40% 60% 100% 4.60
Trainer Overall Effectiveness
Financial Evaluation
Actual savings based on outcomes
Cost of performance prior to training
- Cost of performance after training
= Performance savings
- Cost of Training
= Cost Savings
Return on Investment Ratio (ROI)
= Cost Savings / Cost of Training
Cost Savings for
Grievance Reduction Training
Pretraining Post training
Management time
10 hours/grievance
10 hrs. X 63 grievances = 630
hrs.
10 hrs. X 8 grievances = 80
hrs.
Union Rep’s time
7.5 hrs/grievance
7.5 X 63 grievances = 472 ½
hrs 7.5 X 8 grievances = 60 hrs.
Total Cost
Management time
Union rep’s time
Total
630 hrs X $50 = $31,500.00
472 ½ hrs X $.= $11,812.50
$43,312.50
80 hrs. X $50 = $4,000.00
60 hrs. X $25 = $1,500.00
$5,500.00
Reduction in cost of grievances: 43,312.50 – 5,500.00 = $37,812.50
Cost of training: -32,430.00
Cost saving for the first year $ 5,382.50
Who Is Interested in the Outcome Data
Outcome Data
Reaction Learning Behavior Results
Training Department
Trainer Yes Yes Yes No
Other Trainers Perhaps Perhaps Perhaps No
Training Manager Yes Yes Yes Yes
Customers
Trainees Yes Yes Yes Perhaps
Trainees’ Supervisor Not really Only if no
transfer
Yes Yes
Management No No Perhaps Yes