how well do you measure e-learning impact? · session 101 - industry accepted approaches to...

Session 101 - Industry Accepted Approaches to Measuring e-Learning Impact -Kent Barnett, Founder and CEO, Knowledge Advisors Inc.

Strategies for Measuring the Value of e-LearningAugust 11, 2005

How well do you measure e-Learning impact?

A. Very effectivelyB. EffectivelyC. InconsistentlyD. Not at all

Organizations that Measure Impact

What ROI issues would you like to address today?

Why invest in learning?

The 8 Primary Reasons to Invest in Learning

Increase SalesDecrease CostsIncrease ProductivityEnhance Customer Loyalty

Improve QualityReduce Cycle TimeMitigate RisksEnhance Employee Loyalty

How well do you understand Kirkpatrick’s Four Levels of

Evaluation?A. Very wellB. WellC. A littleD. Not at all

Are you familiar with the Fifth Level – the Phillips ROI Process?

Balanced Scorecard for Learning

Level Level 22

Level Level 33

Level Level 44

Level Level 55

Did they learn?

Do they use it?

Level Level 11

Did they like it?

What were the results?

What’s the ROI?

Industry Accepted ROI Methods

1. Statistical Analysis2. Phillips ROI Process

Learning Analytics Data Source Model

Learning Analytics Framework

Evaluation Methodologies

Learner-basedPost-event surveyFollow-up survey

Manager-basedPost-event surveyFollow-up surveyManager Survey

Business Results

Job Impact

Learning

Satisfaction

Analyst-BasedPost-program surveyFollow-up surveyManager surveyImpact AnalysisValue Optimization

Cost and Complexity

Key Learning Impact Indices

1. Customer Satisfaction Index2. Learning Effectiveness Index3. Job Impact Index4. Business Results Index5. Value Creation Index6. Overall Performance Index

Key Performance Indicators

Employee Development Benefit to Cost Performance ImprovementCriticality to JobApplication to Job

Knowledge GainedTrainer PerformanceCourseware PerformanceEnvironment Satisfaction

3 Step Process to Value Optimization

1. Identify quantifiable organizational objectives

2. Ensure programs significantly impact organizational objectives

3. Implement continuous improvement process

Are learning investments aligned with business objectives?

What is the time to job impact?

This is the time to job impact performance indicator. It lets you know if and when training was applied to the job. It is collected months after the training, automatically by Metrics that Matter. It helps the organization understand when training effects the job, if at all.

Traffic light

The job impact by course indicators clearly show which courses are having the biggest affect on the job. This report stratifies the responses by the percent that fall above or below ranges of performance. Percentages in red are indicators of little or no job impact versus percentages in green represent significant job impact.

Which courses had the biggest impact on the job?

How much more productive are employees?

You can track specific business results, such as productivity. Asking a manager or a participant to estimate productivity once time has passed on the job is a good indicator of learning’s impact to a specific business objective.

What percent of skills were applied to the job?

This indicator provides insight into the amount of new knowledge/skills picked up in the training for each course (or curriculum or program) that the participant directly applied when they went back on the job as the data is collected several months after the training when the participant is back on the job.

What do managers think?

Managers can provide feedback on how the training is impacting their employees.

Monitor Performance

Benchmark across the key performance indicators on your evaluation forms that link back to the 5 levels of learning measurement. Compare the actual performance to goals and to internal or external benchmarks so you properly monitor using a balanced scorecard approach.

Isolate impact of learning to business results?

A tool enabling the learning team to conduct more comprehensive business impact and ROI analysis.

The tool allows you to store multiple business results tied to multiple programs and show trainings impact and ROI.

Industry Standard Balanced Scorecard

The scorecard is a great management tool to showcase value to key stakeholders.

This report shows you each level of learning and the Key Performance Indicators within each level.

Client Case Study

Learning Provider Recognized an Opportunity LP Knew Training was Valued at Client This Case Will:– Explain Report Issued – Provide Reaction by LP and by Client

Large Retailer

Key Client of LPTraining Budget at Risk Significant Investment in Online Learning over Past Three Years

Reporting Process

LP used Metrics that Matter to pull available data into a report for ClientData already existed with no additional collection efforts needed by LP Report included:– Quality Indicators– Effectiveness– Projected Impact– Results– Comments

Effectiveness Indicators

Metrics that Matter showed that learners experienced very high knowledge and skill gains from the training LP provided.

Projected Impact

Metrics that Matter quantified the impact the training had on the job of employees. A 7 to 1 benefit to cost ratio was predicted.

Results

Metrics that Matter quantified the actual results 2 months AFTER the training for an on-the-job analysis showing the ROI actually occurred.

Comments

“Great self-paced content.”“Instructor handled the virtual

session very effectively.”“Everything that was needed was

provided”“Great job. Thank you very much.

Metrics that Matter easily tabulated the comments from the students from this client so LP could report this feedback directly to the client management.

Debrief:

Client Director of Learning– “The ROI was very helpful. We used that

analysis and formatted it for other training we do because the model makes practical sense.”

– “The VP of HR saw this value now we both are presenting this to the CFO and President, thanks!”

Contact

For further information contact:Kent Barnett, CEO+1 312 423 8750kbarnett@knowledgeadvisors.com

www.knowledgeadvisors.com

how well do you measure e-learning impact? · session 101 - industry accepted approaches to...

Documents

introducing our learning & evaluation mandy barnett april...

introducing our learning & evaluation mandy barnett april...

6.7 thomas barnett

algebra barnett

learning impact awards

g barnett webquest

home - oklahoma rural water association · shannon farmer &...

fiscal and macroeconomic impact of privatization · fiscal...

the economic and fiscal contribution of the barnett...

smirp barnett 2002

hudson/barnett group

barnett io 1999

barnett & finnemore

barnett shale characterization

barnett quad avi

barnett exparte mar15_12

steve barnett - hits

barnett ad 2

tro memo (barnett)

barnett play.ppt