1

Patti P. Phillips, Ph.D.205-678-8101www.roiinstitute.net

BOTTOMLINEThe

ROIon

Objectives

ØFivelevelsofevaluationthatserveasthefoundationoftheROIMethodology

ØSixtypesofresultsthatrepresentthechainofimpact

ØTenstepsoftheROIMethodologyprocessmodel

ØTwelveguidingprinciplesthatsupporttheROIMethodology

2

Stayuntiltheend

4

Whyfocusonresults?

Whyfocusonresults?

• Investmentsbecomebusinessdecisions.

• Fundingisprotectedandenhanced.

• Programscancontinueandexand.• Businesspartnershipsaredeveloped.

• Supportincreases.• Seatat“the”tablebecomesavailable.

5

6

Lackofresultsleadsto…

Failuretodemonstrateresultsleadsto:

1. Investments being made on perception not evidence.2. Middle managers ceasing to support investments in learning.3. Diminishing commitment for learning and talent development.4. Waning influence.5. Curtailing of budgets and funding.6. Diminishing team satisfaction.

7

Howdoyoudefineresults?

8

Levels of Evaluation Measurement Focus

0. Inputs and Indicators The input into the project in terms of scope, volume, efficiencies, costs Participants, Hours, Costs, Timing

1. Reaction & Perceived Value

Reaction to the project or program, including the perceived value

Relevance, Importance, Usefulness, Appropriateness, Intent to use, Motivation to take action

2. Learning & Confidence

Learning to use the content and materials, including the confidence to use what was learned

Skills, Knowledge, Capacity, Competencies, Confidence, Contacts

3. Application & Implementation

Use of content and materials in the work environment, including progress with actual items and implementation

Extent of use, Task completion, Frequency of use, Actions completed, Success with use, Barriers to use, Enablers to use

4. Impact and Consequences

The consequences of the use of the content and materials expressed as business impact measures

Productivity, Revenue, Quality, Time, Efficiency, Customer Satisfaction, Employee Engagement

Typical Measures

5. ROI Comparison of monetary benefits from program to program costs

Benefit-Cost Ratio (BCR), ROI%, Payback Period

The ROI Calculation

BCR =Program BenefitsProgram Costs

ROI =Net Program Benefits

Program CostsX 100

BCR =$750,000$425,000

ROI = X 100$750,000 - $425,000$425,000

Try it!

BCR =$750,000$425,000

= 1.76:1

ROI = X 100$750,000 - $425,000$425,000

= 76%

Try it!

ROI Institute and ATD research show that the data CEOs receive are not demonstrating they are getting what they want out of their talent investment. (N=96)

MeasuresCurrently Measure

Should Measure Importance

Inputs and Indicators 94% 86% 6

Efficiency 78% 82% 7

Reaction 53% 22% 8

Learning 32% 28% 5

Application 11% 61% 4

Impact 8% 96% 1

ROI 4% 74% 2

Awards 40% 44% 3

What CEOs Want

2015 Measurement and Metrics Study by CLO Magazine

35.6% Use business data to demonstrate the impact of the training organization on the broader enterprise.

21.6% Use ROI for that purpose.

22.6 % Plan to implement ROI within twelve months.

9.7% Plan implementation in twelve to twenty-four months.

17.3% Plan implementation with no time frame.

(N=335)

71.2%

CLOs Use or Plan to Use ROI

2016 i4cp/ROI Institute study indicates that impact and ROI studies are top of mind when it comes to human capital analytics projects. (N=317)

Impact and ROI are Top of Mind

12%

29%

32%

34%

57%

69%

Other(pleasespecify)...

ForecastingROI(e.g.forecastingtheROIofawellnessinitiative)

Convertingdatatomoney(e.g.jobengagementtomonetaryvalues)

Developingpredictivemodels(e.g.screeningtestpredictsretention)

Developingrelationshipsbetweenvariables(e.g.connectingabsenteeismto…

MeasuringtheimpactandROIofaprogram(e.g.measuringtheROIofleadership…

15

Ultimately, the success of any program is based on whether it improves business results.Training Top 10 Hall of Fame – May 2017

17

ROI Methodology

EvaluationPlanning

DevelopObjectives

ofSolution(s)

DataCollection Reporting

DevelopEvaluationPlansandBaselineData

CollectDataDuringSolutionImplementation

CollectDataAfterSolutionImplementation

CaptureCosts

ofSolution

CalculatingtheReturnonInvestment

IdentifyIntangibles

Develop,Report,andCommunicate

Results

• Reaction and Planned Action

• Learning and Confidence

• Application and Implementation

• Business Impact

• ROI• Input/Indicators

DataAnalysis

IsolatetheEffectsofSolution

ConvertDatato

MonetaryValues

Business Alignment and Forecasting The ROI Methodology

Learning Needs

Preference Needs

Evaluation

Reaction

Learning

Application

Impact

ROI

Reaction Objectives

Learning Objectives

Application ObjectivesPerformance Needs

Impact Objectives Business Needs

Payoff Needs ROI Objectives

End HereStart Here

5

4

3

2

5

4

3

2

1 1Initial Analysis

Input Needs 0 Input Objectives Input0

Program

Alignment Model

0%

10%

20%

30%

40%

50%

60%

1 2 3 4 5 6 7 8

32% 29%

5%

55%

37%

11% 13% 11%

Method1. Control Groups2. Trend Line Analysis3. Forecasting Methods4. Participant Estimates5. Manager Estimates6. Sr. Management Estimates7. Expert Input8. Customer Input* Survey of Users, N = 235

Isolation Techniques

1. Report the complete story2. Conserve evaluation resources3. Use the most credible sources of data4. Choose the most conservative alternatives5. Isolate the effects of the program6. Assume no data, no improvement7. Adjust estimates for error8. Throw out the extreme and unsupported claims9. Use first year benefits for short-term programs10. Include fully-loaded costs11.Report intangible benefits12.Communicate results to all stakeholders

12GuidingPrinciplesensurereliabilityintheprocess.

Reaction

Learning

Application

Impact

ROI

Isolate the Effects of the Program

Intangible Benefits

Inputs

The final story includes six types of data.

VZLU1Leadership Core+Functional

ROI=95%

4.8outof5.0onoverallsatisfactionwithtraining

86%ofparticipantsIncreasedSkillsontheJob

21%

33%

44%

Lunch

Leadership Module

Small Group Exercises

Top3factorsaffectingimpact:

Oneononeswithdirects

ManagerSupport

Opportunitytopractice/apply

40%

20%

10%

Participants =152WORTHIT?

IMPACTEDTHEBUSINESS?

APPLIEDONTHEJOB?

LIKEIT?

CLIMATEFACTORS?

NetAddsaboveQuota*

12.0NetAddsaboveQuotaperpersondirectlyduetotraining

LEARNANYTHING?

98%ofgainednewandvaluableknowledgefortheirjob

*Graphshowstotalimprovementbeforeisolation

Forevery$1wespent,wemadeback$1.95

Used with permission from Verizon

Sample Case StudiesTransnet Coach Business South AfricaSupervisor coaching program focused on improving beat rate and sales targets. ROI = 32.64%

PolyWrigton USAWork engagement intervention focused on controllable waste, rework, and engagement. ROI = 399%

Industrial Company GermanyGlobal sales training focused on improving sales. ROI = 161%

Baptist Hospital USASexual harassment prevention program focused on reduction in complaints, turnover, and bad behavior.ROI = 1051%

What can you do?

Moveyourpracticeforwardby…

25

ü Assessing your readiness for ROI

ü Identifying stakeholders and their data needs

ü Determining the purpose of your evaluation practice

ü Identifying programs suitable for ROI

ü Building capability in the ROI Methodology

FreeTools

26

27

2018 ROI Certification Events•Charlotte, NC – January 15-19

•Atlanta, GA – February 12-16

•Orlando, FL – February 26-March 2

•New Orleans, LA – March 5-9

•Seattle, WA – April 9-13

•Philadelphia, PA – April 23-27

•San Diego, CA – May 7-11

•Denver, CO – May 21-25

•Long Beach, MS – May 21-25

•Atlanta, GA – August 20-24

Visit www.roiinstitute.net/roi-certification for more details

28

Patti P. Phillips, Ph.D.205-678-8101www.roiinstitute.net

BOTTOMLINEThe

ROIon