performance‐focused smile sheets ltem (the learning ... · will thalheimer, phd...
TRANSCRIPT
Will Thalheimer, PhD
Performance‐Focused Smile Sheets&
LTEM (the Learning‐Transfer Evaluation Model)
1. Do you have a philosophy or approach to learning evaluation that you embrace? If so, what is it?
2. What’s the best reasoning you’ve heard for why we should evaluate learning?
Bridging Gap between Research and Practice
Research Translation
https://WorkLearning.com/catalog/
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan University
& Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learn
Learning‐Transfer Evaluation Model
There is no perfect measurement tool!
“Our measures are not perfect, but they should instead be thought of as approximations.”
Deborah L. Bandalos, Author of the 2018 book:
Measurement Theory and Applications for the Social Sciences
“The evaluation of training programs in terms of ‘results’ is progressing at a very slow rate.”
Donald Kirkpatrick1960
“For the most part, the benefits of industrial training have been taken on faith. Few demands have been made to evaluate it in a rigorous manner.”
Ronald BurkeResearcher
1969
“With some notable exceptions… relatively little work has been devoted to making evaluation more useful and worthwhile.”
Robert Brinkerhoff1981
“ASTD’s research revealed that the actual practice of evaluation doesn’t often follow the strict recommendations of evaluation literature.”
1990
Only 20% were able to do the learning measurement they wanted to do…
2007
“In every year [from 2005‐2007], more than 90 percent rated measurement as the number one or number two area they would like to improve.”
Josh BersinPrinciple and Founder
Bersin, Deloitte Consulting LLP2008
Dixon, G., & Overton, L (2017). Making an impact: How L&D leaders can demonstrate value. Available at: www.towardsmaturity.org/impact2017
In general, are you able to do the learning measurement you want to?
The Kirkpatrick Four-Level Model
Level 1
ReactionLevel 2
LearningLevel 3
BehaviorLevel 4
Results
The Kirkpatrick-Katzell Four-Level Model
Read about Raymond Katzell’s role:
https://is.gd/originator
Level 1
ReactionLevel 2
LearningLevel 3
BehaviorLevel 4
Results
How would you rate the typical learner‐feedback questions you’ve seen?
Terrible
1
Bad
2
Okay
3
Good
4Given, that there is NO perfect
measurement…
Smile Sheetsto
Learning
r=.09
Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997).
A meta-analysis of the relations among training criteria.
Personnel Psychology, 50, 341-357.
Smile Sheetsto
Learning
r=.09
Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D.
(2008). A review and meta-analysis of the nomological network of
trainee reactions. Journal of Applied Psychology, 93, 280-295.
1990’s 2000’s
Weak Relationship is below .30 and .09 is VERY WEAK
So…SMILE SHEETS tell us VERY LITTLE about Learning
Likert-like Scales provide Poor DataA. Strongly AgreeB. AgreeC. Neither Agree Nor DisagreeD. DisagreeE. Strongly Disagree
54321
Sharon Shrock and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion‐Referenced Test Development, offer the following wisdom:
On using Likert‐type Descriptive Scales (of the kind that uses response words such as “Agree,” “Strongly Agree,” etc.):
“…the resulting scale is deficient in that the [response words] are open to many interpretations.” (p. 188)
Research shows that learnersdon’t always know their own learning…
Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).
Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).
Learners Fail to Properly Use ExamplesRenkl (1997).
Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).
Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).
Quite simply, the BEST book on smile sheet creation and utilization, Period!
Karl M. KappProfessor of Instructional Technology
Bloomsburg University
Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.
Robert O. BrinkerhoffProfessor Emeritus, Western Michigan University
& Director, Brinkerhoff Evaluation Institute
Evidence‐based practice at the master level.
Julie DirksenAuthor of Design For How People Learn
http://is.gd/TrainingResearch2012
http://is.gd/ddResearch
1. Red‐flagging training programs that are not sufficiently effective.
2. Gathering ideas for ongoing updates and revision of a learning program.
3. Judging strengths and weaknesses of a pilot program to enable revision.
4. Providing instructors with feedback to aid their development.
5. Helping learners reflect on and reinforce what they learned.
6. Helping learners determine what (if anything) they plan to do with their learning.
7. Capturing learner satisfaction data to understand—and make decisions that relate to—the reputation of the training and/or the instructors.
8. Upholding the spirit of common courtesy by giving learners a chance for feedback.
9. Enabling learner frustrations to be vented—to limit damage from negative back‐channel communications.
Modified based on work by Robert Brinkerhoff
Reasons for Smile Sheets
Effectiveness
of the
Learning
Reputation
of the
Learning
Supporting Learners
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
HOW ABLE ARE YOU to put what you’ve learned into practice on the job? Choose One.
A. I am NOT AT ALL ready to use the skills taught.
B. I have GENERAL AWARENESS but will NEED MORE GUIDANCE to put the skills into practice.
C. I need MORE HANDS‐ON EXPERIENCE to be GOOD at using these skills.
D. I am FULLY COMPETENT in using these skills.
E. I am CAPABLE at an EXPERT LEVEL in using these skills.
A Better Smile Sheet Question
0 10 20 30 40 50 60
HOW ABLE ARE YOU to put what you’ve learned into practice on the job? Choose One.
Unacceptable
Acceptable
Deplorable
Superior
Percentage of Respondents
Unlikely
NOT AT ALL READY
HAVE AWARENESS,NEED MORE GUIDANCE
NEED MOREHANDS ON EXPERIENCE
FULLY COMPETENT
COMPETENTAT EXPERT LEVEL
Ultimate Goal Primary Goals Secondary Goals Tertiary Goals
After the course, when you begin to apply your new knowledge at your worksite, which of the following supports are likely to be in place for you? Select as many items as are likely to be true.
A. I will have my PROGRESS MONITORED BY MY SUPERVISOR in applying the learning.
B. I will have someone available TO COACH OR MENTOR ME in applying the learning.
C. I will have easy access to a COURSE INSTRUCTOR to contact for guidance and support.
D. I will have JOB AIDS to guide me in applying the learning to real job tasks.
E. I will be PERIODICALLY REMINDED of key learning concepts/skills over the next few months.
F. I will NOT get much direct support, but will rely on my own initiative.
A Question About Follow-Through…
But won’t learners dislike
these new type of
questions?
Percentage saying NEW QUESTIONS EQUAL OR BETTER:
90%
Percentage saying NEW QUESTIONS
BETTER:
80%
Smile Sheets should be ONLY ONE PARTof our learning evaluation efforts
Smile Sheets
Understanding
Remembering
Motivation to Apply
After Supports
Meeting Target Goals?
Job Performance
Organizational Results
Learner Expectations
Other Expectations
Supports:
Management Support?
Workplace Obstacles?
Reinforcement?
Reminders?
Learners able to:
Understand?
Remember?
Make Decisions?
Apply the Learning?
Traditional Smile Sheets
Performance‐Focused Smile Sheets
eLearning
Self‐StudyLearning Program
Employee Exit Survey
Performance-Focused Smile Sheets:
What Questions Do You Have?
I am clear about what is expected of me as a result of going through this training.
A. Strongly DisagreeB. DisagreeC. NeutralD.AgreeE. Strongly Agree
Good
Okay
Bad
After training, my manager and I will discusshow I will use the learning on my job.
strongly disagree ‐‐‐‐‐ 1 2 3 4 5 6 7 ‐‐‐‐‐ strongly agree
What’s Wrong?
What plans, if any, do you have for talking with your manager in the next 10 days about how you will use the learning in your work?
A. My manager and I have made plans for at least TWO meetings.B. My manager and I have made plans for at least ONE meeting.C. My manager HAS MENTIONED the idea, but we don’t yet have firm plans.D. I will SEEK OUT my manager and ask for at least one meeting on this.E. We are LIKELY TO DISCUSS my use of the learning as we work together.F. It is DOUBTFUL that we will spend much time discussing my use of the learning.
Better!
Why Do We Need a New Evaluation Model?
“The Kirkpatrick framework has a number of theoretical and practical shortcomings.”
“[It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders…” (p. 91)
http://is.gd/TrainingResearch2012
A
D
D
I
E
Assessment
Design
Development
Implementation
Evaluation
The Kirkpatrick-Katzell Four-Level Model
Level 1
ReactionLevel 2
LearningLevel 3
BehaviorLevel 4
Results
Who?
What Messages Does the Four‐Level Model Send?
Messaging of Four-Level Model of Learning Evaluation
Harmful Messages (Sent or Missed)
• Does Not Warn Us Against Ineffective Evaluation Practices
• Ignores the Role of Remembering
• Level 2 Learning is Mashed into One Bucket
Beneficial Messages
• Don’t Just Focus on Learning! Focus on Results too!
• Learner Opinions Are Not Most Important
Special Thanks:
• Julie Dirksen• Clark Quinn• Roy Pollock• Adam Neaman• Yvon Dalat• Emma Weber• Scott Weersing• Mark Jenkins• Ingrid Guerra‐Lopez• Rob Brinkerhoff• Trudy Mandeville• Mike Rustici
Industry Empathy
Grok Problem
Triggering Event
Generate Quick
Solution
Gather Blind‐Spot Feedback
Improve Solution
Reality‐Test
Solution
Understanding‐‐‐‐‐‐‐‐
Remembering
Understanding‐‐‐‐‐‐‐‐
Remembering
Understanding‐‐‐‐‐‐‐‐
Remembering
Work Performance
Performance In Learning
Compared to
Kirkpatrick‐Katzell
Four‐Level Model of Learning Evaluation
Level 1 ‐‐ Reaction
Level 2 ‐‐ Learning
Level 3 ‐‐ Behavior
Level 4 ‐‐ Results
Two Ways to Use LTEM1. Assessing Your Evaluations2. Learning Design & Development:
Working Backward from Your Goals
2018
2018
2019
2020
Two Ways to Use LTEM1. Assessing Your Evaluations2. Learning Design & Development:
Working Backward from Your Goals
1. Sales Increase by 5%
2. Managers Coach Better
3. Simulated Coaching
4. Scenario Questions
5. IF‐THEN Decisions
6. Perf‐Focused Questions
What Questions do You have?
What Questions do You have?
Merrill’s First Principles
What, if anything, have you learned in this session, that you might want to apply to
your ID projects?
• LTEM Report and Model — https://is.gd/LTEM999
• Katzell’s Contribution — https://is.gd/Katzell
• Updated Smile‐Sheet Questions 2018 — https://is.gd/Questions2018
• A Better Net Promoter Question — http://is.gd/replaceNPS
• Be Careful When Benchmarking — https://is.gd/DoNotBenchmark
• Debate About Kirkpatrick Model — http://is.gd/epicbattle
• Better Responses on Smile Sheets — https://is.gd/betterresponses
Subscribe to my Newsletter: — https://www.worklearning.com/sign‐up/
Want to give your
smile sheet a
checkup?
SmileSheets.com
Will Thalheimer, PhD
Phone: +1‐617‐718‐0767Email: [email protected]: WorkLearning.comBook: SmileSheets.comTwitter: @WillWorkLearn
https://www.worklearning.com/contact/
The Decisive Dozenfor Learning Design and Learning Measurement
http://is.gd/ddResearch
1. Content
2. ExposureBaseline
3. Guiding Attention
4. Creating Correct Conceptions
5. Repetition
6. Feedback
7. Variation
Engagement & Understanding
8. Retrieval Practice
9. Context Alignment
10. SpacingRemembering
11. Persuasion
12. PerseveranceApplication