evaluating learning: strategy, math & …...4 john deere | evaluating learning: strategy, math,...

Steve Engelhardt, John Deere Power Systems November 6, 2018

EVALUATING LEARNING: STRATEGY, MATH & RESULTS

John Deere | Evaluating Learning: Strategy, Math, Results| November 20182

Bloom’s Levels of Learning Applied

Create

Evaluate

Analyze

Apply

Understand

Knowledge


Kirkpatrick's Levels of Evaluation

• Measure student reaction to the classSubjective – immediate survey resultsWhat did you think about the training?

Level 1• Measure learning/knowledge retention

Objective – test resultsShow me what you learned.

Level 2• Measure behavior change

Subjective – impact on job performanceDid performance improve after taking the class?

Level 3• Measure business impact

Objective – metrics basedHow did the class help the business?

Level 4


Level 1 Survey

Course Summary1. How likely are you to recommend this course to other students?2. How likely are you to recommend this content to others in a similar job role?3. How likely are you to recommend this delivery method for this course?

The Instructors…1. … seemed to care about my learning.2. … could relate to my job.3. … gave me useful resources to solve

problems.4. … communicated course content in

an understandable way.5. … were knowledgeable in the subject

matter.6. … delivered the training in an

effective manner.

Course Design1. The pace of the course was appropriate for my learning.2. Content applies to me and will help me do my job better.3. Course content as presented in the class met my

expectations.4. The student workbook was a useful tool for my learning.5. The exercises, activities, and examples were effective at

accomplishing the course objectives.6. The instructions for the activities were clear.7. The technology used in the course effectively supported

the learning experience.


xx

xx

xx

xx

Level 1 Reporting Example (older)

Likely to Recommend

Monthly reportingØ Course NPS resultsØ Course AttendanceØ Distribution of CommentsØ Successes and Opportunities

UnLikely to Recommend

Passive

NPS (Net Promotor Score)Ø Who will “Promote” our training?Ø Common to EnterpriseØ Our target = Sample IDT and ILT results

Population

Monthly Reporting NPS (Net Promotor Score)


Level 1 Reporting Example (newer)

50’s 80’s 70’s

vILTs8.xx*

ILT9.xx*

WBTs4.xx**

World Class

Excellent

Good

* = 11-pt scale** = 5-pt scale

11-point scale = 9 & 105-point scale = 5

11-point scale = 7 & 85-point scale = 4

11-point scale = 0 to 65-point scale= 1 to 3

Monthly reportingØ NPS results by Delivery MethodØ Mean Score addedØ Comments Separately reportedØ Successes and OpportunitiesØ Standard NPS Scale


Level 1 Survey Open-Ended Comments

“Good/No Change”

Technology

Course Design

Instructors

Content

Pace/TimeWorkbook

Visuals

13%

7%

17%

2% 47%

5%3%

6%


What are “Drivers” of NPS rating?

(p<.05)=Statistically Significant

Key:

Magnitude=length of line

B(relationship)= slope of line

Key vILT Driver: Content Expectations

Key WBT Driver: Content Applicable

74.5% of the way NPS varies is explainedby the full model of four predictors.

Content ExpectationsMagnitude 10%Relationship (B)=0.535

Appropriate Delivery MethodMagnitude 2.4%Relationship (B)=0.232

Instructor KnowledgeMagnitude 0.4%Relationship (B)=0.132

Content RelevancyMagnitude 0.5%Relationship (B)=0.124

Drivers of NPS Results


1. Assessment Revisionsa) Rewrote Fall/Winter 2017, Live March 2018b) Comprehensivec) Higher cognitive leveld) Demonstrate Skills

2. Assessment Results Reportinga) Pass – Fail ratesb) Assessment Survey Results

3. Tying levels together challenge with GDPR

Level 2 – Assessment Results


Level 2 Assessment- Group Pass RatesVikings Packers Da Bears


– Level 3 Follow-up Survey Pilot Project• Contact students and/or their supervisor 60-90 days after class to ask how the class has impacted

their on-the-job performance

Level 3 Evaluation Did performance improve?Level 3

Impact to businessLevel 4

– Level 4• Analyze existing data (Diagnostic Times, DTAC Calls, Warranty Claims, Customer Satisfaction) to

determine business impact of training

evaluating learning: strategy, math & …...4 john deere | evaluating learning: strategy, math,...

Documents