evaluation of training rationale for evaluation types of evaluation data validity issue evaluation...
TRANSCRIPT
Evaluation of Training
• Rationale for Evaluation
• Types of Evaluation Data
• Validity Issue
• Evaluation Design
Rationale for Evaluation
• Organizational activities should be regularly examined to ensure they are occurring as planned and producing the intended results
• To correct things (people, processes, products or services) that deviate from their objectives
Evaluation Schedule - Sample
Types of Evaluation Data
Two areas of evaluation:Process evaluation: how well the training was designed, developed, and implemented
Outcome evaluation: how well the training achieves its objectives
Process DataBefore Training• Analyzing the processes used to develop
training– effectiveness of needs analysis– assessment of the training objectives– evaluation of the design of the training– assessment of evaluation tools– examination of the training package
• Pre-training evaluation enables correction of errors or omissions
Were needs diagnosed correctly What data sources were used? Was a knowledge/skill deficiency identified? Were trainees assessed to determine their pre-requisite KSAs?
Were needs translated into training objectives? Were all objectives identified Were the objectives written in a clear, appropriate manner?
Was an evaluation system designed to measure accomplishment of objectives?
Was the training program designed to meet all the training objectives? Was previous learning that might either support or inhibit learning in training identified? Were individual differences assessed and taken into consideration in training design? Was the trainee motivation to learn assessed? What steps were taken to address trainee motivation to learn? Were processes built into the training to facilitate recall and transfer? What steps are included in the training to call attention to key learning events? What steps are included in the training to aid trainees in symbolic coding and cognitive organization? What opportunities are included in the training to provide symbolic and behavioural practice? What actions are included in the training to ensure transfer of learning to the job?
Are the training techniques to be used appropriate for each of the learning objectives of the training?
Potential Questions for a Process Analysis: Pre-training
Process Data: During Training
• Examining whether the implementation of the training program reflects what was proposed, designed and included in the training manual
• Assessing the appropriateness of training techniques and methodologies for achieving training objectives
Potential Questions for Process Analysis – During Training
Was there a match between trainer, training techniques, and training/learning objectives?
Were lecture portions of the training effective?Was involvement encouraged/solicited?Were questions used effectively?
Did the trainer appropriately conduct the various training methodologies (case, role play, etc.)?Were they explained well?Did the trainer use the allotted time for activities?Was enough time allotted?Did the trainees follow instructions?Was there effective debriefing following exercises?
Did the trainer follow the training design and lesson plans?Was enough time given for each of the requirements?Was time allowed for questions?
Uses of Process Data• Trainer
– Helps to determine what works well and what does not
• Other trainers– May be able to apply if the process is generalizable
• Training manager– Decision making when the training fails, or problem
with particular trainer
Outcome Data
• To determine how well training has met its goals
• Four types of outcomes that are normally used:– reaction– learning– behaviour– organizational results
Reaction Outcomes
• measures of trainee’s:– perceptions– emotions– subjective evaluationsof the training experience
• the first level of evaluation• favourable reactions are important in
creating motivation to learn
Reaction Outcomes (cont’d)
• the data are used to determine what the trainees thought about the training
• usually it is the only type of evaluation undertaken
• reaction questionnaires are divided into two types:– affective : measures general feelings– utility : beliefs about the value of training
Reaction Outcomes (cont’d)
• do not measure learning – only opinions and attitudes of trainees about the training
• categories normally included in developing a reaction questionnaire:– relevance of training– training content, materials and exercises– trainer’s behaviour– facilities
Reaction Outcomes (cont’d)
• Training Relevance– provides an indication of the value of training– perceived value influences interests
• Training Content, Materials and Exercises– includes written materials, videos, exercises and other
tools of instruction– based on trainees’ feedbacks, modifications could be
made
Reaction Outcomes (cont’d)• Reactions to the Trainer(s)
– evaluations on the trainer’s actions– how well the trainer conducted the training
programme
• Facilities– items related to the facilities – noise, temperature,
seating arrangements etc.– helps in determining whether the facilities are to be
used for future training programmes
Sample Reaction Questionnaire - TrainerSample Reaction Questionnaire - TrainerPlease circle the number that reflects the degree to which you agree or disagree with the following statements. 1 Strongly disagree 2 Disagree 3 Neither agree nor disagree 4 Agree 5 Strongly agree 1 The trainer did a good job of stating the objectives at
the beginning of training 1 2 3 4 5
2 The trainer made good use of visual aids when making presentations
1 2 3 4 5
3 The trainer was good at keeping everyone interested in the topics
1 2 3 4 5
4 The trainer encouraged questions and participation from trainees
1 2 3 4 5
5 The trainer made sure everyone understood the concepts before moving on to the next topic
1 2 3 4 5
6 The trainer summarized important concepts before moving to the next module
1 2 3 4 5
7 Overall, how would you rate this trainer (check one) _____1. Poor; I would not recommend this trainer to others _____2. Adequate; I would recommend this trainer only if no others were available _____3. Average _____4. Good; I would recommend this trainer above most athers _____5. Excellent; this trainer is among the very best I’ve ever worked with
Sample Reaction Questionnaire – Instructor and Materials
Learning Outcomes
• Learning objectives are developed from TNA• The gap between trainee’s KSAs and the
required KSAs defines the learning that must occur
• Three types of learning outcomes:– knowledge– skills– attitudes
Learning Outcomes - Knowledge
• There are three types of knowledge:– declarative– procedural– strategic
Knowledge Outcome - Declarative
• factual knowledge• paper-and-pencil tests are often used to
determine whether trainees have learned the knowledge– easier to administer and score– if properly developed, it accurately measures most
declarative knowledge• multiple-choice test is the most common
– reliable– covers a broader content
Knowledge Outcome -Procedural
• organizing information into mental models • one method commonly used is “paired
comparisons”– trainee’s answers are compared to an
expert’s answers
Knowledge Outcome - Strategic
• deals with the ability to develop and apply cognitive strategies in problem solving
• assesses the level of understanding the trainee has about the decisions and choices he/she makes
• trainees are required to provide the rationale for decisions/choices made
Learning Outcome Questionnaire - Example
Skill-Based Outcomes
• to determine if a skill or set of behaviours has been learned
• it measures the level of learning not whether they are used on the job
• two levels of skill acquisition:– compilation– automaticity
Skill-based Outcome - Compilation
• to determine the additional skills acquired by the trainee from training
• various methods may be used:– structured scenario – multiple raters using standardized methods
Skill-based Outcome - Automaticity
• the speed in which a skill is being used• one method is speed test – performance
has to be completed within a certain time• e.g emergency procedures for pilot
Attitudinal Outcomes
• assessment of changes in attitudes• attitudinal scales are often used• to measure changes, evaluation should
use pre/post measure of responses on a scale
• caution: weaknesses of self-report
Questionnaire - Example
Ambiguity?
Behaviour Outcomes
• to determine whether the training has transferred to the job
• primary sources of data:– interviews– questionnaires– direct observation– performance records
Behaviour Outcomes (cont’d)
• questionnaires are often used, because:– opinions can be obtained from a large number
of employees– the information can be tabulated to yield a
numerical response– respondents are anonymous– short amount of time is required
Organizational Results• The objectives of training are developed to solve
organizational problems• Evaluation is conducted to assess whether the
training has solved the problems• Some aspects of evaluation may include:
– reduction in defects– increase in productivity– reduction in grievances– increase in quality
Validity Issue in Evaluation
• Internal validity:– whether the change was a function of
training– the confidence that the results of the
evaluations are in fact correct
• External validity– whether the same results are
generalisable to other groups of trainees
Threats to Internal Validity
• History– other events that take place concurrently with training
• Maturation– changes that occur because of the passage of time
• Testing– presence in pretest/post-test design that use the
same test
Threats to Internal Validity (cont’d)
• Instrumentation– two different but equivalent tests – is it really
equivalent– could cause differences in the two scores
• Statistical regression– tendency for those who score either very high
or very low on a test to regress to the middle when taking the test again (“regression to the mean”
Threats to Internal Validity (cont’d)
• Initial Group Differences– comparison between trainees and a similar
group of employees (control group who have not been trained)
– it is important that the control group be similar in every way to the training group
• Loss of Group Members– poor scorers in pretest may be demoralised
and drop out of the training
Threats to Internal Validity (cont’d)
• Diffusion of Training– trainees may share the knowledge with the
control group– post-test scores may not show differences
between the training group and the control group
• Compensating Treatment– control group may get special assistant
because they are not given training
Threats to Internal Validity (cont’d)
• Compensatory Rivalry– the control may see the situation as a
challenge and compete for better performance
– post-test scores may not show difference in performance between the two groups
Threats to Internal Validity (cont’d)
• Demoralised Control Group– the control may perceive that they are made
control group because they are not as good as the training group
– they may give up and actually reduce productivity
– post-test scores would show difference but not due to training
External Validity
• The evaluation has to be internally valid before it can be externally valid
• If training is effective for a group, will it also be effective for other groups?
i.e is the evaluation generalizable
Threats to External Validity• Testing
– subsequent training groups may not have pre-tests – difficult to conclude that they would be as effective
– those who took pre-test may have focused on certain materials highlighted in the test
• Selection– a training program with identical design may produce
different results for different categories of employees
Threats to External Validity (cont’d)
• Reaction to Evaluation– success from a group may make further evaluation
unnecessary– those who are evaluated may got more attention
(“Hawthorne Effects”)• novelty• special• received feedback• know they are being observed• inspired by the trainer
Threats to External Validity (cont’d)
• Multiple techniques– effectiveness could be from a combination of
techniques:• lecture – no effect• video instruction – effective
– video instruction is then used for future training, but ineffective
– the effectiveness of training in the first group was the result of both lecture and video instruction