developing metrics that demonstrate the value of technical communications
TRANSCRIPT
May 2011
Developing Metrics that Demonstrate the Value of Technical Communications
Vic Passion
© 2011 TechProse All rights reserved. 2
• What are metrics?
• Kirkpatrick’s levels of evaluation
• The value of the baseline
• Choosing your metrics
• Calculating ROI
• Wrap-up
What we’ll cover …
2
© 2011 TechProse All rights reserved. 3
Measures to demonstrate the value of technical communications
3
What are metrics?
Let’s explore why metrics are so important…
© 2011 TechProse All rights reserved. 4
Depends on:
– Amount and complexity of content
– Type and number of deliverables
– Frequency of updates
– Translation requirements
– Amount of customization
– Tools and techniques
– Format of source material
– Other factors
4
What do technical communications cost?
© 2011 TechProse All rights reserved. 5
• What are metrics?
• Kirkpatrick’s levels of evaluation
• The value of the baseline
• Choosing your metrics
• Calculating ROI
• Wrap-up
What we’ll cover …
5
© 2011 TechProse All rights reserved. 6
Source: *Kirkpatrick, Donald L, Evaluating Training Programs: The Four Levels
Kirkpatrick’s levels of evaluation and objectives
4. Results
3.Transfer
2.Learning
1.Reactions
Performance-based objectives
Satisfaction
Job task performance
Enterprise, division, and departmental objectives
© 2011 TechProse All rights reserved. 7
Who wants to know the results?
4. Results
3.Transfer
2.Learning
1.Reactions
Supervisors, line managers
Technical communications managers
Managers
Department managers, execs, CEO
© 2011 TechProse All rights reserved. 8
A transit provider has unacceptable outages for its subway escalators and elevators. Management has designated documentation funds to help reduce time out of operation. What would you measure for at each level?
Business case
4. Results
3.Transfer
2.Learning
1.Reactions
© 2011 TechProse All rights reserved. 9
• What are metrics?
• Kirkpatrick’s levels of evaluation
• The value of the baseline
• Choosing your metrics
• Calculating ROI
• Wrap-up
What we’ll cover …
9
© 2011 TechProse All rights reserved. 10
Support Calls
0
50
100
150
200
250
300
350
Q1 Q2 Q3 Q4
Quarter 2007
Nu
mb
er o
f su
pp
ort
cal
ls
Compelling metrics show the delta between the baseline and the post-documentation data
The value of the baseline
Documentation and website launched in Q3
© 2011 TechProse All rights reserved. 11
• Build all deliverables around what you want end users to be able to do, integrated with corporate strategy
• Define what they can do before the release—the baseline
– Number of errors
– Revenue generated
– Orders processed
– Stock shipped
– Stock returned
– Etc.
The ABCs of performance-based metrics
© 2011 TechProse All rights reserved. 12
• You get what you measure
• Easy to capture…
– Performance after release (but what was it before?)
– Performance of groups with most access to documentation vs. groups with least access (but what about complexity of tasks?)
– Improvement/degradation over time (but what about upgrades?)
• Characteristics of good metrics
– A baseline
– Near-term benefits
– Meaningful, tangible
– Careful analysis
Choosing the right metrics
© 2011 TechProse All rights reserved. 13
• What are metrics?
• Kirkpatrick’s levels of evaluation
• The value of the baseline
• Choosing your metrics
• Calculating ROI
• Wrap-up
What we’ll cover …
13
© 2011 TechProse All rights reserved. 14
• Break into groups.
• Read your case study and decide what you would measure to build a compelling case.
• Choose a group leader to present your measures.
Building a compelling case
© 2011 TechProse All rights reserved. 15
Business cases
#1: After three severe injuries to machine operators, a national paper products company requires all machine
operators and managers to receive training on an updated user’s guide.
#2: A technology company delivers training to its employees and partners on a new enterprise software system for quoting and ordering.
#3: After a tax season in which the software company is unable to keep up with support calls, the company launches an online knowledge base.
#4: After getting feedback from clients and suppliers that translated materials and updates to documentation are taking too long, an international online retailer implements single sourcing.
10 mins
© 2011 TechProse All rights reserved. 16
Wrap-Up
© 2011 TechProse All rights reserved. 17
• What are metrics?
• Kirkpatrick’s levels of evaluation
• The value of the baseline
• Choosing your metrics
• Calculating ROI
• Wrap-up
What we’ll cover …
17
© 2011 TechProse All rights reserved. 18
Calculating ROI
Example: Reduce transit technician repair time by 30 minutes per repair
30 min x $100/hour = $50
$50 x 1,500 repairs per year = $75,000
Cost of documentation = $50,000
(Benefit – Cost)/Cost = ROI
($75,000-$50,000)= $25,000 savings
$25,000/$50,000 investment = 50% ROI
© 2011 TechProse All rights reserved. 19
– Use conservative assumptions
– Focus on only one or two compelling metrics
– Keep it simple
– Document your assumptions
Building a case for a technical communications program
© 2011 TechProse All rights reserved. 20
• Break into the same groups.
• Using the same case study, calculate the ROI.
• Document your assumptions.
• Choose a group leader to present your ROI.
Calculating ROI
© 2011 TechProse All rights reserved. 21
Business cases
#1: After three severe injuries to machine operators, a national paper products company requires all machine
operators and managers to receive training on an updated user’s guide.
#2: A technology company delivers training to its employees and partners on a new enterprise software system for quoting and ordering.
#3: After a tax season in which the software company is unable to keep up with support calls, the company launches an online knowledge base.
#4: After getting feedback from clients and suppliers that translated materials and updates to documentation are taking too long, an international online retailer implements single sourcing.
15 mins
© 2011 TechProse All rights reserved. 22
Wrap-Up
© 2011 TechProse All rights reserved. 23
• Don’t take your budget for granted
• Use higher levels of evaluation to speak to execs in your organization
• Capture the baseline before measuring
• Capture meaningful, tangible metrics with near-term benefits
• Estimate proposed ROI by:
– Documenting your assumptions clearly
– Using conservative assumptions
– Focusing on only one or two compelling metrics
– Keeping it simple
• (Benefit – Cost)/Cost = ROI
Key points to take home
© 2011 TechProse All rights reserved. 24
Vic PassionTechProse
Walnut Creek, CA, USA925.956.4231
Your turn!
© 2011 TechProse All rights reserved. 25
Helping clients navigate change
www.techprose.com
© 2011 TechProse All rights reserved. 26
Backup
© 2011 TechProse All rights reserved. 27
Business case #1
After three severe injuries to machine operators, a national paper products company requires all machine operators and managers to receive training on an updated users guide.Goals:
– Fewer lawsuits against the company this year
– Lower employee absenteeism
– Lower insurance costs
© 2011 TechProse All rights reserved. 28
Business case #2
A technology company delivers training to its employees and partners on a new enterprise software system for quoting and ordering. Goals:
– More efficient use of employee and partner time
– Better retention of partners
– Ability to see customers with expiring contracts
© 2011 TechProse All rights reserved. 29
Business case #3
After a tax season in which the software company is unable to keep up with support calls, the company launches an online knowledge base.
Goals:
– Faster response time
– Better retention of end users
– Lower technical support costs
© 2011 TechProse All rights reserved. 30
Business case #4
After hearing from clients and suppliers that translated materials and updates to documentation take too long, an international online retailer implements single sourcing. Goals:
– Shorter release cycle
– Better retention of end users
– Lower technical support costs