srp developing a quantitative model to measure training effectiveness

8/12/2019 SRP Developing a Quantitative Model to Measure Training Effectiveness

1/13

Symbiosis Institute of Business Management, Pune

Title Developing a quantitative model to measure training effectiveness

Author Shailendra Singh Bhati

Nida Shahid

Daniel Picardo

Source Insights Inside

References

1. Basarab, David / Root, Darrell - The Training Evaluation Process: A Practical

Approach to Evaluating Corporate Training Programs (Evaluation in Education

and Human Services)

2. Rae, Leslie- Assessing the Value of Your Training: The Evaluation Process from

Training Needs to the Report to the Board

3. Phillips, Jack / Stone, Ron- How to Measure Training Results : A Practical Guide

to Tracking the Six Key Indicators

4. Kirkpatrick, Donald D / Kirkpatrick, James D - Evaluating Training Programs:

The Four Levels

5. Zielinski, Dave Evaluating Trainings Impact (The Complete New Training

Library, Vol. 9)

6. Coulthard, Glen J Critique of Kirkpatricks Four Levels of Evaluation; Purdue

University

7. Bernthal, Paul- Methods for calculating ROI and bottom-line impact; DDI

Center for Applied Behavioral Research

8. Naish, Richard ROI Evaluation: Building the business case for people

development

9. Arthur, Winfred Jr / Edens, Pamela S / Bell, Suzanne T Effectiveness of

Training in Organisations; Texas A&M University / USAF Research Library

10.Fox, Paul G Measuring Training Results; Fox Performance Training


2/13


Document TypeSecondary Research Paper

Subject Human Resources

Date November, 2006

Abstract

This paper describes the techniques of measuring the return on investment on the training

function of HR in an organization. The various models of evaluating ROI with their

respective criticisms and the proposed training evaluation model emphasize the business

need for a quantitative model in this HR function.


3/13


Exposition:

EXECUTIVE SUMMARY

The main objectives of our research were:

To examine the reasons for the ineffectiveness of traditional models of trainingevaluation

To examine the feasibility of contemporary metrics like return on investment (ROI)

as a tool to measure training effectiveness

To develop (using an Excel spreadsheet) a dynamic model which will help HR

professionals calculate thepotential ROI on a proposed training programme before an

employee is sent for the same

INTRODUCTION

The evaluation of training is essentially desirable to every HR/ Training professional. But

organizations usually relegate this to the background and it tends to become a short-term

priority shadowed by long-term business objectives.

While happy sheets are useful in evaluating training in a crude form, they seldom hold

much bargaining power with stakeholders who fund training at the expense of other

organizational needs. They are more interested in demonstrated results i.e. measures of

how training expenditures contribute to organizational growth.

Some of the major arguments in favour of evaluating training are:

To validate training as a business tool that an organization can use to improve its

performance and profitability.

To justify the costs incurred in training, by thorough quantitative analysis so as to

avoid training budget cuts when cash flows are scarce in an organization

To help improve the design of training programmes to provide better value and


4/13


increased benefits for an organization.

To help in selecting alternative approaches including a variety of classroom, on-job

and self-study methods using comparative evaluation techniques

Different organizations specify criteria on which they wish to evaluate their training

programs. These vary from numbers, direct cost, indirect cost, efficiency, performance to

schedule, income received, reactions, learning, behavioural change and performance

change. Regardless of the method, one element remains common among all these; they

wish to measure the impact of training with credibility and consistency.

THE KIRKPATRICK MODEL: A CRITIQUE

Introduced more than four decades ago, Kirkpatricks sequential process for evaluating

training programs is well-known, well-respected, and well-criticized in both academia

and the corporate community. In his book Evaluating Training Programs, Kirkpatrick

(1998) clarified his four-level evaluative progression, in which each successive level

builds upon the previous one and provides even more information.

The American Society of Training and Development (ASTD) recently reported that over

60% of organizations that evaluate their training programs currently use the Kirkpatrick

model. Of surveyed organizations, the ASTD reports that 77 percent evaluate participant

satisfaction and reactions (Level One), 38 percent measure learning (Level Two), 14

percent evaluate behavioral change and transfer (Level Three), and only 7 percent

measure results (Level Four) or business impact. Reasons for the decreasing participation

rates include increasing levels of difficulty and higher costs (financial, labor, and time)

associated with implementing the upper evaluative levels.


5/13


General criticisms

Lack of detail

The model is too simplistic and errs in suggesting that evaluations can be definitive,

replicated, and generalized to larger groups (Galloway, 2005). Some researchers also

attempt to re-classify the framework as a taxonomy rather than a model (Stoel, 2004).

Overemphasis on the training program

Leading this criticism is Fred Nickols, who promotes an alternative approach for

evaluating training that shifts the focus onto the stakeholders. Nickols argues that the

Kirkpatrick model is so wrapped up in assessing the effectiveness of the training program

that it loses sight of more important objectives and audiences. The basic premise of the

stakeholder approach is that several groups in the organization (trainers, training

managers, funding managers, trainee supervisors, trainees, vendors, and developers) have

a stake in the success of the training and, therefore, should have a say in its design,

development, and evaluation because the purpose of an evaluation is to make a value

judgment, the analysis needs to place each stakeholders information requirements,

perceptions of value, contributions (costs), and inducements (benefits) at the center of the

evaluation. And, only when each stakeholders needs are met is the evaluation complete.

Primarily outcome focused

While outcomes are certainly important, this criticism argues that Kirkpatricks model

fails to adequately consider why certain results occur and instead focuses solely on

what happens (Boverie, Mulcahy, & Zondlo, 1994). In Level One questionnaires, for

example, participants have reported what they thought the trainer wanted to hear, rather

than their true thoughts and feelings (Boverie et al.).In another study, participants tended

to underestimate their pre-training skills and overestimate their post-training skills in

an attempt to help justify the training (Boverie et al., p. 3). While the results of these

questionnaires and assessments show positive outcomes, they are of limited use to

trainers, developers, and managers.


6/13


Lack of business focus

Kirkpatrick has also been accused of focusing too much on learners and too little on the

core business objectives. While Jack Phillips attempts to field this criticism by adding

Level Five, ROI, to the framework (Stoel, 2004), detractors still point to the limited use

of business methodologies, tools, and metrics in the evaluation process. Organizations are

extremely cost- conscious and decision-makers demand formal cost analyses using

techniques and reporting formats that they understand. Consider that the first three

evaluative levels focus on the participant: Level One captures feelings, Level Two

assesses learning, and Level Three observes the participant in the workplace. Only in

Level Four does Kirkpatrick focus in on the business.

Implied correlations where research proves otherwise

One of the most documented criticisms is Kirkpatricks implied correlation between

reaction, learning, and changed behavior in the workplace. The basis for this criticism is

Kirkpatricks position that, in order to achieve behavioral change in the workplace,

learners must react favorably to the training program and learn the required knowledge,

skills, and attitudes. Unfortunately, research has shown that there is little correlation

between a learners reaction to training and learning, and even less between reaction and

performance or behavioral change (Alliger, Tannenbaum, Bennet, Traver, & Shotland,

1997;Ruona et al.,2002

Ignored importance of support systems

In the same vein, Brinkerhoff and Dressler (2002) argue that Kirkpatricks focus on

changed learner behaviors blocks consideration of the importance of the workplaces

managerial and support system. They write, training produces only capability, not

performanceany number of performance system factors can, and often do, derail the

transformation of learning into performance results (Brinkerhoff & Dressler, 2002, p.

16).

Specific level criticisms

Reaction level


7/13


This level is not indicative of the training's return on investment, as it does not measure

what new skills the learners have acquired nor what they have learned that will transfer

back to their working environments. This has caused some evaluators to downplay its

value. However, the interest, attention and motivation of the participants are critical to the

success of any training program. People learn better when they react positively to the

learning environment.

Learning level

Measurements at level two might indicate that a program's instructional methods are

effective or ineffective, but it will not prove if the newly acquired skills will be used back

in the working environment.

Behaviour change level

It is important to measure behavior because the primary purpose of training is to improve

results by changing behavior. New learning is no good to an organization unless the

participants actually use the new skills, attitudes or knowledge in their work activities.

Since level three measurements must take place after the learners have returned to their

jobs, the actual level three measurements will typically involve someone closely involved

with the learner, such as a supervisor. Although it takes a greater effort to collect this data

than it does to collect data during training, its value is important to the training

department and organization. Behavior data provides insight into the transfer of learning

from the classroom to the work environment and the barriers encountered when

attempting to implement the new techniques learned in the program.

Results level

As we move from level one to level four, the evaluation process becomes more difficult

and time-consuming, although it provides information that is of increasingly significant

value. Perhaps the most frequently used measurement is level one because it is the easiest

to measure. However, it provides the least valuable data. Measuring results that affect the

organization is more difficult and is conducted less frequently, yet yields the most

valuable information...whether or not the organization is receiving a return on its training


8/13


investment. Each level should be used to provide a cross set of data for measuring

training program.

OTHER THEORIES: A CRITIQUE

Control group method

This approach has high validity in terms of isolating the effects of the training

programme. An experimental design is used; one group participates on a training

programme and another group does not. Care is taken to ensure the two groups are

similar in all other respects, using such techniques as randomization.

However the disadvantage is that some people do not get the benefit of the training

programme. Also the cost of keeping tabs on the performance of two groups control

group and trained group is very high

Modelling method

This approach is more analytical and mathematical than simply drawing a trend line,

since rather than a linear solution; there may be some type of non-linear relationship.

When several variables influence the results, sophisticated statistical models need to be

used. However this model requires too much data and availability of appropriate data is

difficult resulting in a less than accurate prediction of performance variables

Estimation approach

This approach asks participants how much of their performance improvement was due to

the training programme. This estimate is then adjusted for the level of confidence they

have in their estimation. Thus if a participant believes that 60% of the improvement was

due to the training but is only 70% confident of this, the real improvement is calculated

as 42% (60%*70%). The estimations made by the learners are preferable since they

should be the most aware of the reasons for improvement in their performance.

However this is rather subjective since only trainees viewpoint is considered. (To add

validity to these estimates, supervisors, colleagues or direct reports can also be asked. By


9/13


weighting each of these estimates, an average 360-degree estimate is calculated) The

disadvantage of this method is that it is only an estimate and the input data may be

unreliable since participants may find it hard to come up with percentage improvements

and percentage confidence levels. However, it is an inexpensive and quick method, which

produces data that can be easily understood by HR and the board.

BUSINESS NEED FOR QUANTITATIVE MODEL

Making business decisions entails a wide range of factors and involves intricate tradeoffs.

In todays knowledge economy, training expenditure invariably forms a major chunk of

the organizations budget. The top management therefore needs to know the effectiveness

of these training programs and their contribution to the top line of the company. The

decision to send an employee on a particular training program depends mainly on the

increase in effectiveness of an employee at a particular job and thus an increase in his

contribution towards the organizations growth. Therefore to be able to understand and

gauge the importance of training program it is very important to express its effectiveness

in money terms. Such a model would help the training manager to sell a training

program to the top management and the line manager to nominate only those employees

who would benefit most out of such training programs.

No organization has the budget to do all the good things it wants to do. With a

quantitative model for training effectiveness it would be possible for the top management

to rank various training programs and go ahead with only those which are critical to the

business and promise maximum returns. Such a model would enable the management to

know whether the returns from a training program outweigh the costs incurred on it. The

quantitative model would enable the top executive to align the training programs to the

vision of the company and allow them to increase shareholder value.


10/13


PROPOSED MODEL

Why post-training evaluation does not work

Post-training evaluation, which is usually carried out at the end of a classroom session

can be compared to a process of inspecting finished goods as theyre passing out of the

factory gate for delivery to customers. Some of the reasons why post-training evaluations

continue to feature at the end of training programmes are:

Maintenance of training records as a part of the overall compliance and quality

system requirements

Lack of a better method to satisfy the training department and managers who

nominated candidates for the training programme, as well as a means to justify

investment

A greater emphasis on what happens at a training programme, rather that

competency behaviours exhibited before and after training

We feel a strong need for a pre-training evaluation model which will measure potential

return on training before an employee can be nominated for a training programme.

The primary advantages of such a model are:

It uses direct inputs from the firms competencies, the potential candidates pastappraisal forms, training needs identification and personal development plan

It reduces unnecessary expenditure as the benefits to the trainee can be evaluated

before the training is imparted

It eases the decision making process for training department as well as the

department manager with regards to nomination of employees for a training

program

It helps the top management to budget and prioritize training programs keeping in

view the return on investment and the effectiveness


11/13


Process flow for proposed model

Review job

evaluation forms

Classify the job into

specific tasks

Prioritize the job

tasks as critical,

normal and minimal

Assign the

percentage of

compensation to

each task based on

relative importance

Determine current

and desired level of

efficiency on the task

for which training is

proposed

Run benefits and

programme costs on

the ROI calculator

Is ROI greater than

100%?

Do not send

employee to the

training program

Nominate

employee for the

training program


12/13


Proposed training evaluation model

Value added by training = (S*T) (P2-P1)

Where, S = Salary, T = Time (% of total work time), P1 = Pre-training productivity (% ),

P2 = Post-training productivity

Example of a filled in ROI calculator

Section Line Sub section Explanation Calculator

Line 1 Current total salaryIncludes total benefits less

statutory deductions500000

Line 2% of time spent on tasks for

which training is sought

To be filled by potential

candidate35%

Line 3 Importance of task1=critical, 0.5=normal,

0.1=minimal0.7

Task compensation

(current value)

Line 4Weighted training related task

compensationLine 1 x Line 2 x Line 3 122500

Line 5 Current effectiveness (%)

Based on feedback from

supervisor, appraisal forms

and assessment centres

40%

Line 6Desired level of effectiveness

(%)From job description 60%

Effectiveness

Line 7Desired increase in

effectiveness (%)

(Line 6 - Line 5)/Line 5 50%

Line 8Value added to task

compensation (post-training)Line 7 x Line 4 61250

Task compensation (future

value)

Line 9Value of new task

compensationLine 8 + Line 4 183750

Line 10 Per diem cost of training Monetary value (Rs.) 10000

Line 11 Number of days of training Use fraction if less than a day 2

Line 12 Total training cost Line 10 x Line 11 20000Cost of training

Line 13 Training cost as a percentageof current total salary

Line 12/ Line 1 4.0%

Line 14 Net potential ROI Line 8 Line 11 41250Potential ROI

Line 15 Net potential ROI (%) (Line 8 Line 11)/ Line 11 206%


13/13


Explanatory notes

Line 1 Current salary

As the model is based on the compensation for each task classified from the job

description, it necessary to included the total salary of the potential candidate. Here, total

salary should not be confused with cost-to-company (CTC). Total salary is taken to

mean, CTC less all statutory deductions

Line 2 - % of time spent on task for which training is sought

Line 3 Importance of task

The potential candidate or the reporting manager can fill in these sections. If filled in by

the manager, then the ideal time to be spend on the task (as listed in the job description/

job evaluation) must be used

Line 4 Weighted training-related task compensation

This refers to the portion of compensation that is paid out the candidate to perform the

task on which training is sought. The allocation is purely notional and would not form a

part of the candidates pay slip or annual salary revision.

Line 8 Value added to task compensation (post-training)

This is amount by which task compensation would increase as a result of increased

effectiveness due to training. This value clubbed with the original task compensation

value would provide the new task compensation (Line 9)

Line 15 Net potential ROI

If the net potential ROI is more than 100% then it is profitable to nominate and send the

candidate for the training program, given the investment that will be incurred to conduct

the program. If less that 100%, the candidate could be put on hold until a cheaper training

program is announced or till the need for a higher new increase in level of effectiveness

(Line 7) is felt.

srp developing a quantitative model to measure training effectiveness

Documents