practical strategies for learning evaluation – metrics that matter in identity & passport...

Practical strategies for Learning Evaluation – Metrics That Matter in

Identity & Passport Service (IPS)

Prepared by : Andrew Somiah

Systems Training Manager

Date 18th July 2011

Version 2.0

Not protectively marked

1

Overview of IPS

Identity and Passport Service (IPS) is a Public Sector

Organisation

Approximately 3800 employees spread across the

UK

Core Business Products:

─ Passports & Identity Management

─ Birth, Marriage, Death & Civil Partnership

Certificates

2

Overview of Learning & Development Function in IPS

Centralised Learning & Development function with a

wide geographical remit.

Classroom trainer-led events

E-learning delivered by Kallidus Learning

Management System (LMS)

Blended learning solutions managed via the Learning

Management System (LMS).

Approximately 589.6 days of training delivered for

2009-2010 financial year

5 Levels of Evaluation Model used in IPS (Jack Philips Model)

Learning Measurement & MTM Strategy

• MTM is integrated with the IPS LMS– Estimate, Isolate & Adjust technique used by MTM

• All training courses including some e-learning are evaluated

• Internal targets and an Organisational Development Score Card has been set to drive strategy

• Three key indicators that are monitored monthly:

– Trainer Performance

– ROI: Cost Benefit Ratio

– Learning Impact & Value (business results)

• IPS targets are incorporated into KPOs for L&D staff who are performance managed with these

5

Learning Measurement & MTM Strategy

Implementing

Tactical

Eff

ectiv

enes

s

Time

IPS-Strategic

Optimized

1

2

3

4

5

Collecting evaluation data and measure business results

LMS dashboards presentation to business

Presenting monthly analyses to business

Comprehensive

6

1st Quarter of 2010 Figures

April May June

No of Events 85 84 90

No of Delegates 322 395 650

Training days 59.9 79.9 38.5

Average delivery per trainer (Target: 100 days per yr i.e. 9.0 average per month)

2.7 6.3 5.4

Cost benefit Ratio (Target: £1:3.00)

£ 1:1.74 £ 1:1.86 £ 1:2.44

Average Trainer Score (Target: 6.25)

5.41 6.37 6.43

Evaluation Returns (Target: 65%)

52.02% 58.08% 56.78%

7

Impact on Business

• To improve impact and value to business: information is displayed on the LMS dashboard

8

Practical Approach to Data Analysis

9

Evaluation Data Analysis

Review & Analysis Cycle

10

• Human Capital ROI Card

• Business Results ROI Card

• Instructor Performance Summary Report

• Training Improvement Summary Report by Trainers

• Project Initiation Documents/Action plan by Training Managers

• Course Effectiveness Summary Report (Director & board Level)

• Talent Development Executive Report (Director & board Level)

11

Reports Used for Practical Analysis

Challenges & Tips

• There was initial resistance when the system was implemented as it was a cultural change

• IPS has put in place an Organisational Development balance score card accepted at board level

• The internal benchmark targets have been incorporated into the organisations Key Performance Indicators

• Align evaluation strategy to business objectives

•Note: Its is a Process not a project expect results over time.

• Simple continuous marketing of the benefits to business units helps improve evaluation return rates

12

Summary

• Aspirations- Evaluation to become second nature and part of the culture

• MTM is providing an invaluable resource for:

– demonstrating the worth of training to organisation

– provides performance information for making crucial strategic decisions

– Could have a positive impact on the bottom line of any organisation

Questions

Contact and Further Information

Andrew Somiah

Systems Training Manager

Identity and Passport Service

HQ Learning & Development

4th Floor Mailpoint B NW Peel Building

2 Marsham Street

London

SW1P 4DF

T: 0207 0356 798

E: andrew.somiah@ips.gsi.gov.uk

Stuart Halliday

EMEA Sales Director

KnowledgeAdvisors

P: +44 (0) 203 371 1165

M: +44 (0) 771 7507 945

shalliday@knowledgeadvisors.com