practical strategies for learning evaluation – metrics that matter in identity & passport...
TRANSCRIPT
Practical strategies for Learning Evaluation – Metrics That Matter in
Identity & Passport Service (IPS)
Prepared by : Andrew Somiah
Systems Training Manager
Date 18th July 2011
Version 2.0
Not protectively marked
1
Overview of IPS
Identity and Passport Service (IPS) is a Public Sector
Organisation
Approximately 3800 employees spread across the
UK
Core Business Products:
─ Passports & Identity Management
─ Birth, Marriage, Death & Civil Partnership
Certificates
2
Overview of Learning & Development Function in IPS
Centralised Learning & Development function with a
wide geographical remit.
Classroom trainer-led events
E-learning delivered by Kallidus Learning
Management System (LMS)
Blended learning solutions managed via the Learning
Management System (LMS).
Approximately 589.6 days of training delivered for
2009-2010 financial year
5 Levels of Evaluation Model used in IPS (Jack Philips Model)
Learning Measurement & MTM Strategy
• MTM is integrated with the IPS LMS– Estimate, Isolate & Adjust technique used by MTM
• All training courses including some e-learning are evaluated
• Internal targets and an Organisational Development Score Card has been set to drive strategy
• Three key indicators that are monitored monthly:
– Trainer Performance
– ROI: Cost Benefit Ratio
– Learning Impact & Value (business results)
• IPS targets are incorporated into KPOs for L&D staff who are performance managed with these
5
Learning Measurement & MTM Strategy
Implementing
Tactical
Eff
ectiv
enes
s
Time
IPS-Strategic
Optimized
1
2
3
4
5
Collecting evaluation data and measure business results
LMS dashboards presentation to business
Presenting monthly analyses to business
Comprehensive
6
1st Quarter of 2010 Figures
April May June
No of Events 85 84 90
No of Delegates 322 395 650
Training days 59.9 79.9 38.5
Average delivery per trainer (Target: 100 days per yr i.e. 9.0 average per month)
2.7 6.3 5.4
Cost benefit Ratio (Target: £1:3.00)
£ 1:1.74 £ 1:1.86 £ 1:2.44
Average Trainer Score (Target: 6.25)
5.41 6.37 6.43
Evaluation Returns (Target: 65%)
52.02% 58.08% 56.78%
7
Impact on Business
• To improve impact and value to business: information is displayed on the LMS dashboard
8
Practical Approach to Data Analysis
9
Evaluation Data Analysis
Review & Analysis Cycle
10
• Human Capital ROI Card
• Business Results ROI Card
• Instructor Performance Summary Report
• Training Improvement Summary Report by Trainers
• Project Initiation Documents/Action plan by Training Managers
• Course Effectiveness Summary Report (Director & board Level)
• Talent Development Executive Report (Director & board Level)
11
Reports Used for Practical Analysis
Challenges & Tips
• There was initial resistance when the system was implemented as it was a cultural change
• IPS has put in place an Organisational Development balance score card accepted at board level
• The internal benchmark targets have been incorporated into the organisations Key Performance Indicators
• Align evaluation strategy to business objectives
•Note: Its is a Process not a project expect results over time.
• Simple continuous marketing of the benefits to business units helps improve evaluation return rates
12
Summary
• Aspirations- Evaluation to become second nature and part of the culture
• MTM is providing an invaluable resource for:
– demonstrating the worth of training to organisation
– provides performance information for making crucial strategic decisions
– Could have a positive impact on the bottom line of any organisation
Questions
Contact and Further Information
Andrew Somiah
Systems Training Manager
Identity and Passport Service
HQ Learning & Development
4th Floor Mailpoint B NW Peel Building
2 Marsham Street
London
SW1P 4DF
T: 0207 0356 798
Stuart Halliday
EMEA Sales Director
KnowledgeAdvisors
P: +44 (0) 203 371 1165
M: +44 (0) 771 7507 945