“Learning engineering”: The Art of Using Learning Science at Scale to Li7 Performance
October 2015
2
• Kaplan University • Kaplan Legal Education • Kaplan Professional
Education • KU Nursing
• Kaplan Higher Ed – Europe • Kaplan Professional – Europe
• Kaplan Higher Ed – Asia • Kaplan Professional – Asia • Kaplan Higher Ed – Australia • Kaplan Professional –
Australia • In Country Pathways – China • BEO - HO
Kaplan education spans domains and geography
Kaplan University Group
Kaplan Test Prep Kaplan Performance Solutions
Kaplan Asia Pacific Kaplan United Kingdom • Kaplan Int’l Colleges
• Global Pathways • Dublin Business School
Kaplan International Colleges
• KTP Grad • KTP Pre-College • KTP Med • KTP Nursing • KTP Bar Review • Dev Boot Camp • KTP International
3
We know a lot about how expertise works
4
This allows for a “learning engineering” approach
Cognitive Task Analysis
5
From expertise to instructional design
Evidence- Based Instructional Design
Cognitive Task Analysis
6
How exper)se gets acquired with prac)ce
Stage Implications for Instructional Design
7
How exper)se gets acquired with prac)ce
Stage Implications for Instructional Design
Declarative Clear information displays, e.g., job aids, examples, reference material
8
How exper)se gets acquired with prac)ce
Stage Implications for Instructional Design
Declarative Clear information displays, e.g., job aids, examples, reference material
Procedural Build varied Practice tasks, and rich feedback/coaching
9
How exper)se gets acquired with prac)ce
Stage Implications for Instructional Design
Declarative Clear information displays, e.g., job aids, examples, reference material
Procedural Build varied Practice tasks, and rich feedback/coaching
Automated Repeated frequent practice to build speed and accuracy
10
Knowledge Component
Practice/Assessment S
uppo
rtive
Kno
wle
dge
Research gives us clearer guides to help with hard outcomes
11
Knowledge Component
Practice/Assessment
Procedure Decide when to use; perform the steps
Sup
porti
ve K
now
ledg
e Research gives us clearer guides to help with hard outcomes
12
Knowledge Component
Practice/Assessment
Procedure Decide when to use; perform the steps
Sup
porti
ve K
now
ledg
e Fact Recall fact in task context
Research gives us clearer guides to help with hard outcomes
13
Knowledge Component
Practice/Assessment
Procedure Decide when to use; perform the steps
Sup
porti
ve K
now
ledg
e Fact Recall fact in task context
Concept Classify, identify or generate examples and non-examples
Research gives us clearer guides to help with hard outcomes
14
Knowledge Component
Practice/Assessment
Procedure Decide when to use; perform the steps
Sup
porti
ve K
now
ledg
e Fact Recall fact in task context
Concept Classify, identify or generate examples and non-examples
Process Identify causes of faults in a process; predict events in a process
Research gives us clearer guides to help with hard outcomes
15
Knowledge Component
Practice/Assessment
Procedure Decide when to use; perform the steps
Sup
porti
ve K
now
ledg
e Fact Recall fact in task context
Concept Classify, identify or generate examples and non-examples
Process Identify causes of faults in a process; predict events in a process
Principle Decide if principle applies; predict an effect; apply principle to solve a problem, explain a phenomenon or make a decision
Research gives us clearer guides to help with hard outcomes
16
We also know more about mo)va)on. . . Motivation Beliefs
• Value • Self-Efficacy • Attribution • Mood
Motivation Actions
• Starting • Persisting • Mental Effort
Learning/Results
• Practice results • Test results • Fluency/ease • Work results
Self-Efficacy
Lear
ning
Effo
rt
High Moderate Low
17
And about specific ways to make lessons work better
Principle Description Effect size (s.d. units)
Multimedia Use relevant graphics and text to communicate content 1.5
Contiguity Integrate the text nearby the graphics on the screen – avoid covering or separating integrated information
1.1
Coherence Avoid irrelevant graphics, stories, videos, media, and lengthy text 1.3
Modality Include audio narration where possible to explain graphic presentation 1.0
Redundancy Do not present words as both on-screen text and narration when graphics are present
.7
Personalization Script audio in a conversational style using first and second person 1.3
Segmenting Break content down into small topic chunks that can be accessed at the learner’s preferred rate
1.0
Pre-training Teach important concepts and facts prior to procedures or processes 1.3
Etc. Worked examples, self-explanation questions, varied-context examples and comparisons, etc.
??
Source: E-learning and the Science of Instruction, Clark and Mayer, 3rd ed., 2011
18
The impact is not small!
50% 1 sd
84%!
19
We know there is much poten)al to improve*
*Example: Ratings from Kaplan Way for Learning checklist, applied to 9 Kaplan products
20
Have to be careful – what we think is “good” may not be
• Comparison of course teacher view vs. independent teachers’ markings
0"
0.5"
1"
1.5"
2"
2.5"
3"
3.5"
4"
4.5"
course"teacher" Teacher"1" Teacher"2"
Based on 10 randomly selected papers from a writing course
21
The evidence also shows our intuitions aren’t the best guides LSAT Logical Reasoning example
0"
1"
2"
3"
4"
5"
6"
7"
Study"8"worked"examples"
Study"15"worked"examples"
Use"Exis>ng"Kaplan"On"Demand"Instruc>on"
Test"Only"–"No"instruc>on"
N 153* 148* 107 84 Time (mins) 8.15 12.8 99.32 NA
* Significant difference from “No Instruction”
22
All this changes how courses should be developed
Read, Write, Discuss • Outcomes and content not precisely aligned • Limited demonstrations, worked examples,
and practice • General assessment rubrics • High reliance on discussion boards
Existing courses
Prepare, Practice, Perform • Outcomes and content aligned • One lesson per objective • Demonstrations and worked examples • Practice, feedback before assessment • Detailed scoring guides • Less discussion/more practice • Standard instructor materials • Monitoring and support for motivation
Redesigned courses
23
Result: much greater student success
• 11% higher success rate
• 28% increase
• Students in redesigned courses were 1.6 times more likely to be successful
Wald Chi-Square: 10.42, df=1, n=895, Sig<.001.
39%
50%
20%
30%
40%
50%
60%
70%
Control Pilot
Adj
uste
d st
uden
t suc
cess
rate
Adjusted student success rates with 95% confidence limits
24
Faculty supports do help students – but need to check
60%
65%
70%
75%
80%
85%
90%
Attendance Number of assignments*
LMS minutes*
Ret
entio
n
Control Treatment
* Improved learning outcomes
§ §
N: 653 498 625
Impact of faculty dashboards on first year college social studies course
25
At scale, we can pursue mul)ple lines of inves)ga)on
Adaptive
SmartBook
MyLab
Assessment & CLA
CLA Re-Scoring
Assessment
Learning Strategies
Self-Explanation
Self Summary
Self Test
Media Principles
Advanced Organizers
Coherence
Continuity
Modality
Multimedia
Redundancy
Motivation & Self
Efficacy
Judgments of Learning
Motivational Priming
Self Efficacy
Social Norming
Badging
Iconographs
Worked Examples
Early
Later
Others
Section Size
Orientation
Faculty Dashboard
Open Courses
Challenge Exams
26
Running dozens of trials requires careful tracking
1 Study In Prep
Adaptive Learning (Mcgraw Hill Connect AB219)
3 Studies In Development
Attribution I (SOFI) Standing Out Fitting In (AB/MT140) Indigo Live Seminar Delivery (AB/MT203)
19 Studies In Term
Adaptive Learning (McGraw-Hill Connect CM107) Adaptive Learning (SOOMO PS124) Attribution II (KU160) CLA Anchor Paper (PA106) Competency (CS113) Campus LearnLab (CM220 Blended)
Mind Set Challenge (CM220) PERTS Mind Set Challenge (MM150) Science Center Evaluation Study (SC156) AS Section Size (PS124) Self Regulated Learning (CE100) Indigo Live Seminar Delivery (GB512) StereoType Threat Surveys (CJ100, CJ101, MM150, MM207, MM212, MM255)
8 Studies In Assessment (5 Analysis in Progress; 2 Data Requested; 1 Awaiting Data Availability)
Attribution I (HS100 Pre-Intervention) Harvard Study Supporter (CM107 and MM150) CLA Blind Scoring (CJ101 and GB512) Faculty Dashboard (SS310) Social Norming (HU300, MM150, IT301) Worked Examples (CJ130)
34 Studies Completed Advanced Organizers (HN144) CLA Reliability CLA Assignment Order (CJ100) CLA Training (CS204) Orientation Facilitation Bus Section Size Seminar Delivery (LS) Stereotype Threat Historical Review Library Use Historical Review Judgments of Learning (HS101) V4 Learning Platform (IT133) Math Center Feedback (MM207) Motivational Priming (HS100 and PS124) Self Test (HS200) Adaptive Learning (SmartBook HS120) VoiceThread (PA205 and PA201) Worked Examples (14 instances)
27
It’s real work to alter how a large number of IDs build. . .
Online Course
Wor
ksho
p
Analysis Micro Design Development
-8 0 1 3 2 4 5 6 7 8 9 10 11 12
Deliverables
Coaching
Production
Week 13 14 15 16
Delivery
2hr 2hr 2hr 2hr
POs Specs Draft Scripts
C C C
Bug List
Moodle
Revise
Rebuild
Team review / preview ...
2hr
Cohort review ...
Final Scripts
Author
2-month online self-study (40 hours)
2-day workshop Team Projects
4 months coaching
Transfer to products
28
But it matters if you’re after good “learning engineering”
29
Where to find out more?
• LocaAon of course on using (and downloading) the checklist:
hHp://goo.gl/f1RCAu
• Bror’s Blog for more on “learning engineering”:
hHp://www.kaplan.com/brorsblog
• Contact me: [email protected]
30
April 20, 2015 Why We Need Learning Engineers Chronicle of Higher Education