does your learning solution solve business performance problems?
TRANSCRIPT
What is value?
• Think about a training program that you recently implemented.
• How would you define the value delivered by the program?
New definition of value
Be balanced, with qualitative and quantitative data
Contain financial and nonfinancial perspectives
Reflect strategic and tactical issues
Satisfy all key stakeholders
Be consistent in collection and analysis
Be grounded in conservative standards
Come from credible sources
Do we have a place in the table?
• Do training and development professionals participate in business decisions?
• Do training and development professionals participate in government strategic decisions?
Shifting Paradigms
Activity Based Results Based No business need for the program Program linked to specific business needs
No assessment of performance issues Assessment of performance effectiveness
No specific measurable objectives Specific objectives for behavior and business impact
No effort to prepare program participants to achieve results
Results expectations communicated to participants
No effort to prepare the work environment to support transfer
Environment prepared to support transfer
No efforts to build partnerships with key managers
Partnerships established with key managers and clients
No measurement of results or benefit-cost analysis
Measurement of results and benefit-cost analysis
Planning and reporting is input focused Planning and reporting is outcome focused
Levels of Evaluation
10
Levels of Evaluation Measurement Focus
0. Inputs and Indicators The input into the project in terms of
scope, volume, efficiencies, costsParticipants, Hours, Costs, Timing
1. Reaction & Perceived
Value
Reaction to the project or program,
including the perceived value
Relevance, Importance, Usefulness,
Appropriateness, Intent to use,
Motivation to take action
2. Learning
Learning to use the content and
materials, including the confidence to
use what was learned
Skills, Knowledge, Capacity,
Competencies, Confidence, Contacts
3. Application &
Implementation
Use of content and materials in the work
environment, including progress with
actual items and implementation
Extent of use, Task completion,
Frequency of use, Actions completed,
Success with use, Barriers to use,
Enablers to use
4. Business Impact
The consequences of the use of the
content and materials expressed as
business impact measures
Productivity, Revenue, Quality, Time,
Efficiency, Customer Satisfaction,
Employee Engagement
Typical Measures
5. ROIComparison of monetary benefits from
program to program costs
Benefit-Cost Ratio (BCR), ROI%,
Payback Period
Executive view – the report
MeasureWe currently measure this
We should measure this in
the future
My ranking of the importance of this
measure
1. Inputs: “Last year, 78,000 employees received formal learning.” 94%
2. Efficiency: “Formal learning costs $2.15 per hour of learning consumed.”
78%
3. Reaction: “Employees rated our training very high, averaging 4.2 out of 5.”
53%
4. Learning: “92% of participants increased knowledge and skills” 32%
5. Application: “At least 78% of employees are using the skills on the job.”
11%
6. Impact: “Our programs are driving our top 5 business measures in the organizations.”
8%
7. ROI: “Five ROI studies were conducted on major programs yielding an average of 68% ROI.”
4%
8. Awards: “Our learning and development program won an award from American Society of Training and Development.”
40%
Executive view – the need
MeasureWe currently measure this
We should measure this in
the future
My ranking of the importance of this
measure
1. Inputs: “Last year, 78,000 employees received formal learning.” 94% 85%
2. Efficiency: “Formal learning costs $2.15 per hour of learning consumed.”
78% 82%
3. Reaction: “Employees rated our training very high, averaging 4.2 out of 5.”
53% 22%
4. Learning: “92% of participants increased knowledge and skills” 32% 28%
5. Application: “At least 78% of employees are using the skills on the job.”
11% 61%
6. Impact: “Our programs are driving our top 5 business measures in the organizations.”
8% 96%
7. ROI: “Five ROI studies were conducted on major programs yielding an average of 68% ROI.”
4% 74%
8. Awards: “Our learning and development program won an award from American Society of Training and Development.”
40% 44%
Executive view – the gap
MeasureWe currently measure this
We should measure this in
the future
My ranking of the importance of this
measure
1. Inputs: “Last year, 78,000 employees received formal learning.” 94% 86% 6
2. Efficiency: “Formal learning costs $2.15 per hour of learning consumed.”
78% 82% 7
3. Reaction: “Employees rated our training very high, averaging 4.2 out of 5.”
53% 22% 8
4. Learning: “92% of participants increased knowledge and skills” 32% 28% 5
5. Application: “At least 78% of employees are using the skills on the job.”
11% 61% 4
6. Impact: “Our programs are driving our top 5 business measures in the organizations.”
8% 96% 1
7. ROI: “Five ROI studies were conducted on major programs yielding an average of 68% ROI.”
4% 74% 2
8. Awards: “Our learning and development program won an award from American Society of Training and Development.”
40% 44% 3
Chain of Impact
Reaction & Planned Action
Learning
Application & Implementation
Isolate the Effects of the Program
Intangible Benefits
Impact
Matching evaluation levels
Improve work group productivity by 20% in 6 months 4
Initiate at least three cost reduction projects in 15 days 3
Achieve an average cost reduction of $20,000 per project 4
Use counseling skills in 90% of situations where work habits are unacceptable 3
Be able to describe the four elements of the employee assistance program 2
Provide a 4 out of 5 rating on relevance of course content 1
Achieve a 2:1 benefit-to-cost ratio one year after the new leadership program is implemented
5
For each objective listed below, indicate the level of evaluation at which the objective is aimed.1. Reaction and planned action2. Learning3. Application4. Impact5. Return on Investment (ROI)
Key Alignment Questions
Potential ROI ROIPayoffs Objectives5 5
Is this a problem worth solving?
Is there a potential pay off?
What is the actual ROI?
What is the BCR?
Needs ProgramAssessment Objectives Evaluation
Needs ProgramAssessment Objectives Evaluation
Business Impact BusinessNeeds Objectives Impact
Key Alignment Questions
4 4
What is the specific measure?
What happens if we do nothing?
Which business measure improved?
How much is related to the program?
Needs ProgramAssessment Objectives Evaluation
Job Performance Application ApplicationNeeds Objectives
Key Alignment Questions
3 3
What is occurring or not occurring on the job that influences the business measure?
What has changed?
Which skills/knowledge has been applied?
Skills/Knowledge Learning LearningNeeds Objectives
Needs ProgramAssessment Objectives Evaluation
Key Alignment Questions
2 2
What skills or knowledge is needed to support the job performance need?
What did they learn?
What are the takeaways?
Preferences Reaction Reaction &Objectives Planned Action
Needs ProgramAssessment Objectives Evaluation
Key Alignment Questions
1 1
How should the solution be structured?
What was the reaction to the program?
How will the program be implemented?
Business Alignment process
1. Determine payoff needs
2. Identify business needs
3. Identify performance needs
4. Identify learning needs
5. Set Objectives
6. Design for results
7. Measure the business impact
8. Isolate the effects of the program
Training with the end in mind
• What is the value of your training?
– ROI
– Impact
– Application
– Learning
– Reaction
– Input
• Report the whole picture
Build up Capabilities in ROI
• ROI Certification program
– Sydney from 7th to 11th December 2015
– http://insources.com.au/index.php/roi/roi-certification
Feedback Counts!
Your feedback helps ATD 2015 APC continue to provide top-notch educational programs that help you stay on top of a changing profession.
Evaluation forms for this session are available via the mobile app and at the following link: https://zh.surveymonkey.com/r/ATD2015APC