five myths about measuring training’s...
TRANSCRIPT
© 2015 CEB. All rights reserved Delivered: 3 November 2015
Shawn Overcast Managing Consultant CEB | Metrics That Matter Managing Consultant
CEB Metrics That Matter®
Five Myths about Measuring Training’s Impact Why They’ll Get You into Trouble
© 2015 CEB. All rights reserved Delivered: 3 November 2015
The World’s Leading Learning Analytics Software
Key metrics
SmartSheet evaluations
Optimization roadmap
>1 billion data points
Integrated in reporting
Filterable
Scrap learning
Business results
ROI
Built-In Expertise
Learning Impact Benchmarks
Automated Insights
from
© 2015 CEB. All rights reserved Delivered: 3 November 2015 1
30+ Years of Experience
110+ Countries Represented
6,000+ Participating Organizations
300,000+ Business Professionals
Who We Are CEB is the world’s leading member-based advisory company. We have a unique view into
what matters—and what works—when capitalizing on drivers of business performance. With 30 years of experience working with top companies to share, analyze, and apply proven
practices, we begin with great outcomes and reverse engineer to help you unlock your full potential.
As a result, our members achieve outsized returns by more effectively optimizing talent investments, creating new sources of efficiency, reducing risk, and enabling and accelerating
growth.
90% of the Fortune 500
85% of the FTSE 100
76% of the Dow Jones Asian Titans
www.cebglobal.com © 2015 CEB. All rights reserved.
Best Practices & Decision Support Talent Management
Leadership Councils Market Insights Talent Acquisition Talent Development Talent Strategy & Analytics
© 2015 CEB. All rights reserved Delivered: 3 November 2015 4
1 Level 1 Evaluation Results Provide Valuable Information 2 Self-Reported Data are Biased and Lack Value 3 Statistical Significance Testing is Necessary 4 Customization is More Valuable than Standardization 5 Evaluation Begins After Training is Deployed
Five Myths: Insights & Tips to Counter
© 2015 CEB. All rights reserved Delivered: 3 November 2015 5
#1: Level 1 Evaluation Results Provide Valuable Information
Five Myths: Insights & Tips to Counter
© 2015 CEB. All rights reserved Delivered: 3 November 2015 6
Change Your Paradigm. An evaluation can do more than Level 1.
ACTION: Level 1 is No Longer Level 1
Smile Sheet Smart Sheet
If you are going to bother the learners to ask them about their reactions to training, then ask them the really important questions. Ask more than Level 1…
• Did you learn new knowledge and skills?
• Will you be able to apply them? • Will your performance improve
due to training? • Will you have managerial
support? • Will your improvement impact
the performance of the organization?
© 2015 CEB. All rights reserved Delivered: 3 November 2015 7
Change Your Paradigm An evaluation can do more than Level 1.
ACTION: Level 1 is No Longer Level 1
Reaction Prediction
Change the purpose. Don’t confirm that they liked training …
Use the data provided by leaners to predict performance improvement!
I did learn I will apply I will improve So will the company
© 2015 CEB. All rights reserved Delivered: 3 November 2015 8
#2: Self-Reported Data are Biased and Lack Value
Five Myths: Insights & Tips to Counter
It’s just someone’s
opinion
My boss needs highly accurate
statistical results, not opinion data Where is the
scientific rigor?
© 2015 CEB. All rights reserved Delivered: 3 November 2015 9
Various Levels of Value
Type of Data Value Drawbacks
Anecdotal • Rich qualitative information • Details about value
• Not easy to summarize • Not statistically reliable
Self-report • Quantitative and qualitative • Easy to gather on surveys • Can standardize to allow
analysis across courses • Can create predictive
models
• Requires rigor to develop valid and reliable instruments
• Opinion-based and biased
Business / Operations data
• Non-biased • Shows actual business
performance • Outcome measures
specific to individual courses/programs
• Difficult to extract from systems • Requires business analyst • Takes months/years to observe
the outcome • Inconsistent data quality,
accuracy comprehensiveness
© 2015 CEB. All rights reserved Delivered: 3 November 2015 10
Rely on the Wisdom of Crowds • People who know their roles and their skills provide accurate
estimates of how much their performance will improve Trust but verify • Whether you build your own data collection instruments or use
those created by others, apply appropriate rigor to determine if the survey tools are valid and reliable
ACTION: Crowds and Rigor
© 2015 CEB. All rights reserved Delivered: 3 November 2015 11
#3: Statistical Significance Testing is Necessary
Five Myths: Insights & Tips to Counter
If it’s not sig, it doesn’t matter.
Is there a sig relationship or
not?
‘Burden of Proof’?
© 2015 CEB. All rights reserved Delivered: 3 November 2015 12
What is Statistical Significance?
Statistical significance testing is a way of determining if the observed data occurred by chance.
Lear
ner
Per
form
ance
Impr
ovem
ent
Low Manager Support
High Manager Support
x x
x x
x x
x x
x x
x
x x
x x
x
x x
x x
x
Is there a relationship between two or more variables?
Are there differences between groups?
Lear
ner P
erfo
rman
ce Im
prov
emen
t
Scrap Learning
x
x
x
x x
x x
x
x
x
x
x
x
x
x
y = mx + b + e
© 2015 CEB. All rights reserved Delivered: 3 November 2015 13
! Determine for your teams and stakeholders what differences are meaningful—5%, 10% 15%? What difference is practical and would spur action?
! Be cautious: Large data sets can create false positives (small differences are sig) and small data sets can create false negatives (large differences that are not sig)
! The CEO wants to know “What is the impact of training?” ! Educate the C-suite that timely, valid, reliable smart sheet data are
equally as valuable as statistically significant results from impact studies ! Impact studies—time consuming, costly, and resource intensive ! Scalable Smart Sheets—reliable, timely indicators of program quality
ACTION: Balance Statistical and Practical Significance
© 2015 CEB. All rights reserved Delivered: 3 November 2015 14
#4: Customization is More Valuable than Standardization
Five Myths: Insights & Tips to Counter
Content varies per course
Stakeholders often demand questions about certain aspects of the course
Evaluation questions should vary as well
Custom questions provide deep insights for stakeholders
© 2015 CEB. All rights reserved Delivered: 3 November 2015 15
• The use of a core set of questions for every course • Some variations are necessary for learning methodology (e.g., ILT,
SPWB, Virtual ILT) and possibly business unit
ACTION: Incorporate Standards
• Simplicity in design; reduce time and effort • Allows for benchmarking and comparison of results across courses
• Governance: Reduce or eliminate library of evaluation forms • Prevent the deletion of core questions
• Allow for additional questions in an “Additional Questions” section on a limited basis (e.g., only for strategic, visible and costly programs)
Standardization is …
Benefits are …
© 2015 CEB. All rights reserved Delivered: 3 November 2015 16
#5: Evaluation Begins After Training is Deployed
Five Myths: Insights & Tips to Counter
© 2015 CEB. All rights reserved Delivered: 3 November 2015 17
Planning is essential to good evaluation.
1. When forecasting the value of a training program before it is designed
2. When selecting training methods and tools that will maximize learning while minimizing costs
3. While demonstrating the effectiveness of training after it has been deployed
4. While making decisions about future versions of the program, especially in comparison to other programs in the curriculum
ACTION: Ongoing Evaluation
© 2015 CEB. All rights reserved Delivered: 3 November 2015
Key Takeaways
© 2015 CEB. All rights reserved Delivered: 3 November 2015
Thank You
Shawn Overcast
727-452-2478
Danny Brown Account Manager, CEB 321-474-9220 [email protected]
Managing Consultant, CEB