the real roi of leadership training - measureup … · further, when it comes to measuring the...

Evaluating Training Impact Level 1 -6 | Paul Leone, Ph.D. | 2013

HOW TO MEASURE THE ROI OF LEADERSHIP TRAINING


Page | 1

he only way training is ever going to be perceived as a profitable business

imperative, instead of a costly distraction, is if training professionals and

their organizations measure their results. While most training

organizations and vendors recognize the need and urgency to report the impact of

their contributions, the overwhelming majority just don’t end up doing it (or just

don’t do it right). Further, when it comes to measuring the powerful effects of

leadership training, the numbers get even more dismal and the impact studies just

about disappear. This study not only measures the results of a leadership program

from Level 1 (reaction) to Level 5 (Return on Investment), but also introduces a

Level 6 (Transfer Climate) which identifies

the environmental factors that help or

hinder a learner from applying their new

knowledge and skills back on the job.

In this study, the training measured was a

popular leadership program that was

deployed organization-wide at a major

credit card company. Over a period of 12

months, it was launched globally to over 10,000 managers and director-level people

leaders. The sample size for this particular study was 1974 participants across many

diverse regions, lines of business, and staff groups.

The Evaluation Strategy

The evaluation and overall impact of the leadership development efforts were

measured in part using the traditional Kirkpatrick/Phillips 5-level approach and then

T


Page | 2

a sixth level was added to identify the critical factors that can influence the impact of

the training back on the job (see table below).

Measurement Method of Evaluation

Level 1: Reaction Learner responds to survey upon completion of learning event (satisfaction with experience)

Level 2: Knowledge Learner responds to survey upon completion of learning event (new knowledge and skills acquired)

Level 3: Behavior Assessment completed by self and direct reports 3 months after learning event

(observed improvement in leadership skills)

Level 4: Impact Assessment completed by self, and direct reports 3 months after learning event (improved

productivity of direct reports)

Level 5: ROI Cost vs. Benefit analysis based on increase in sales productivity of direct reports over 3 months

Level 6: Transfer

Climate

Assesses factors in participant’s work environment (climate) that will help or hinder the transfer of

learning

Gathering the Data

Participants attending the one-day instructor-led program were given a 5-minute

survey immediately after training. This survey captured satisfaction with the

experience (Level 1) and the level of new knowledge and skills acquired (Level 2).

Three months after the training was completed, participants were then e-mailed a

link to take another survey online and were instructed to invite their direct reports

to complete a similar survey. For participants, this 15-minute survey captured

behavior change (Level 3) and performance

improvements. For direct reports, the survey

included questions about their leaders’ behavior

change and also about how those changes

affected their own performance and


Page | 3

productivity metrics (Level 4). Using these performance increases, a return on

investment (ROI) analysis (Level 5) was then conducted for all respondents

reporting “sales revenue” as their primary performance measure. Also, included in

participants’ self-assessment was a transfer climate index which asked participants

about the work environment they returned to post-training (Level 6).

Results

Level 1 and Level 2

As far as how well the program was received (Level 1) and the amount of learning

that took place (Level 2), participants took no more than 5 minutes to answer a few

key questions about their experience. The results of the quantitative questions are

shown below. One important thing to remember is that most organizations never

get past these Level 1 or 2 measures and often use these rather limited metrics to


Page | 4

extrapolate the ultimate value and potential impact of the program. As you will see

when we get into higher levels of impact, these very first levels become far less

relevant and in no way predict the ultimate outcome and ROI of a training program.

They are useful however to get some preliminary data on how a program is being

received and some important feedback on how to improve the delivery of content.


Page | 5

Level 3

When it came to behavior change (Level 3), we looked at the percentage of

participants who were demonstrating observed levels of improvement. Using a 6-

point improvement scale, we asked both participants and their direct reports to

rate the level of improvement they observed in 6 critical, training-related

behaviors (this number can vary and is primarily defined by the business). One

important thing to look for in these results is similarity in the response

distributions. That is, did direct reports corroborate the self-ratings? Also note

that “high improvement” was defined as the self and direct reports rating a

participant’s improvement either “significant” or “exceptional” on our scale. As

you’ll see shortly, we use this group to calculate higher levels of impact. Below

are our overall response distributions:


Page | 6

Level 4 Of course the next question is “so what…?” What do high improvements in

certain key leader behaviors actually mean to the business? More specifically,

how does improving your leadership behaviors actually affect your direct reports’

levels of productivity? This brings us to our Level 4 measure - Business Impact.

Here we separated our entire sample of direct reports into two distinct groups:

Group A: direct reports of “high improvement” leaders vs. Group B: direct

reports of “no improvement” leaders. When we looked at the extent to which

each of these two groups of direct reports improved their primary measures of

performance, some significant and quite astounding differences emerged. The

direct reports of “high improvement” leaders were increasing productivity by an

average of 48%, while the direct reports of “no improvement” leaders showed

increases of only 11%. See graph below:

While this makes an incredible case for how the behavior of a leader can impact

productivity, the next step was to isolate just how much of this difference in


Page | 7

productivity can be directly attributed to the training. After all, there may be a

multitude of factors that can either make an employee perform better or make a

leader improve his/her behavior over a three-month period.

To isolate the unique influence of the leadership program, we factored in how

much each direct report was influenced by their leaders’ specific changes

(expressed as a percentage)

and then how much each

leader’s changes were

influenced by the program

(expressed as another

percentage).

That is, on our direct report

survey, we asked how much

of their productivity

increases over the past 3

months could be attributable

to improved leadership (e.g. more direction and support from their leaders) and

on our participant survey we asked how much of the improved leadership could

be attributable to the training experience. We then multiply these two

percentages for our Level 4 calculation and that gives us “total productivity

improvement directly attributable to the training”. See next diagram for Level 4

calculation:


Page | 8

From these results we saw that direct reports increased their productivity by an

overall average of 7% as a direct outcome of their leaders attending the

leadership training.

Level 5 – ROI

So what does a 7% increase in productivity mean in dollar

value? To calculate the ROI, we looked at all the employees

reporting Sales revenue as their primary productivity metric.

According to the business, a sales specialist contributed

approximately $50,000 in revenue over a three month period. If an employee’s

contribution was to increase by 7% due to better direction and support from

his/her leader, the added benefit to the company would be approximately $3500.


Page | 9

Now consider that each leader in the study impacted not one employee, but had

an average of 6 employees that were being directly influenced every day. That

brings the total benefit for the company to $21,000 (3500 X 6). If the benefit is

$21,000 in additional revenue, and the cost of the training was approximately

$2,000, the return on investment is 950%. See formula and calculation below:

Level 6 - Transfer Climate

Which factors in the participant’s immediate work environment were predictive

of how much training would be applied and sustained on the job? To measure

this we used the results of our “transfer climate index” attached to the 3-month

post-program survey. Here, we essentially correlated the scores on the transfer

climate questions to the scores on the questions about behavior change and

performance improvements. When we looked at the results of this analysis, we

found 3 environmental or climate factors that were consistently predicting


Page | 10

whether a learner would have “no improvement” or “high improvement” back

on the job. These factors were:

Their manager follows up with them and has one-on-one meetings after the event to discuss

what was learned and how to apply

Their manager endorses the training and supports their development

Their manager specifically recognizes and rewards the improved leadership behaviors

The illustration below depicts just how much these factors could discriminate

between levels of impact. For example, if you look at the first set of data on the

left, 74% of learners in the “high improvement” group were having one-on-one

conversations about how to apply their training, while only 13% of the “no

improvement’ group were having these same conversations. In fact, all 3 factors

were present and pervasive for the “high improvement” group, while they were

all significantly lower for the “no improvement” group.


Page | 11

Conclusion

When it comes to ROI, the training that can (and should) pack the biggest punch is

your leadership training. When leaders increase their effort and improve their

leadership behaviors on the job, they not only become more productive, but they

have a tremendous, positive ripple effect on all the employees they lead and

influence every day. If captured and measured correctly, the “benefit” of these

leader improvements can result in some stunning “returns”. I call this the

“Leadership ROI”.

For this particular study, it was shown that a significant

number of leaders improved their leadership behaviors

and were doing a few crucial things different after the

training. These changes by the leaders clearly made their

immediate direct reports more productive on the job. In fact, these direct reports,

on average, showed a 7% increase in productivity in the 3 months after their leader

attended the training. Each leader also had an average of 6 immediate direct reports

who were being influenced every day by his or her leadership. That meant for every

ONE leader that attended the training and did a few things different afterwards,

there were SIX employees that were enhancing their performance and improving

their primary business metrics. This led to a final ROI for each participant of 950%.

This is the power of the Leadership ROI.

What our evaluation also told us was that participants were clearly taking their

learning back to very different work environments, and these often disparate

“climates” can have a profound effect on how they apply the training. That is, those

who had tactical support from their immediate leader and were deliberately


Page | 12

rewarded for their training-related changes were demonstrating far greater

improvement and impact than those who had less supportive environments.

Further, the results of this Level 6 analysis told us which specific climate factors

were having the biggest quantifiable influence on our training outcomes and ROI.

Imagine the power of knowing what your top factors are and very precisely targeting

them for future training roll-outs and initiatives. Consider how much you could

improve your training’s impact and ROI. This is the power of evaluation. This is

the power of measuring to Level 6.

From these results, it should be clear that measuring and reporting the impact of

your leadership training is not only achievable, but also quite simple if you create

the right strategy and design. In total, the

time I asked from participant’s for this

entire evaluation was no more than 20-25

minutes. While I spent some more time

behind the scenes getting to understand the

behaviors and business metrics we needed

to include, it all still added up to a small

drop in the bucket compared to the time and resources an organization spends

launching any one given training program. By investing just a little in your

measurement strategy, and asking a few simple questions, to the right employees, at

the right times, you can end up a few months later telling an incredible story of

bottom-line impact. How’s that for an ROI?

the real roi of leadership training - measureup … · further, when it comes to measuring the...

Documents