evidence-based thinking about learning and...
TRANSCRIPT
2
• Integrating the design, building, monitoring, and improvement of learning
environments; individualize learning experiences using our scale; and,
ultimately, drive greater student career success.
• Former CLO for K12, Inc. – structured use of technology, cognitive
science, on-line and off-line materials for 1,700 teachers, 55k students
• Former Publisher and General Manager for DK Multimedia, Inc.
• Management consultant with McKinsey & Company
• Education:
- Ph.D. in Electrical Engineering and Computer Science from MIT
- M.D. from Harvard Medical School
- M.A. in Electrical Engineering and Computer Science from MIT
- M.A. in Mathematics from Oxford University
- B.S. in Electrical Engineering and B.S. with Honors in Mathematics
from the University of Washington
Bror Saxberg Chief Learning Officer, Kaplan, Inc.
3
• Kaplan University
• Kaplan Legal Education
• Kaplan Professional
Education
• Nursing
• Kaplan Continuing
Education
• KNEXT
• KTPA
• Kaplan Tutoring
• Kaplan Bar Review
• Kaplan Publishing
• Kaplan Higher Ed – Europe
• Kaplan Professional – Europe
• Kaplan Higher Ed – Asia
• Kaplan Professional – Asia
• Kaplan Higher Ed – Australia
• Kaplan Professional –
Australia
• In Country Pathways – China
• Franklyn Scholar
• Carrick Education
• Global Knowledge Solutions
Kaplan education spans domains and geography
Kaplan University
Group
Kaplan Higher
Education Campuses
Kaplan Test Prep
Kaplan Asia Pacific Kaplan United
Kingdom • Kaplan Int’l Colleges
• Global Pathways
Kaplan International Colleges
4
What Our Students Told Us They Want from a World’s Best Educator
Promise
Pillars
Definitions
We strive to make
education as
personalized to you
as possible−tailoring
our courses around
your individual
needs.
We are dedicated
to getting you the
results that matter
in the time that
matters.
We move quickly
with constant
innovation to
better meet your
needs.
We are here to
help you achieve
success at critical
milestones along
your educational
journey.
10
Agenda
• What evidence says about learning
• What this means for the design of instruction
• What happens when you do this for real
• [How to get outcomes aligned with real expertise]
• [“Teaching & Learning in the 21st Century” – thoughts]
11
Agenda
• What evidence says about learning
• What this means for the design of instruction
• What happens when you do this for real
• [How to get outcomes aligned with real expertise]
• [“Teaching & Learning in the 21st Century” – thoughts]
12
Much research to guide us
Learning
Events (hidden - inside
students’ minds)
Student
Performance (observable -
indicates
knowledge)
Instructional
Events (in the learning
environment)
Knowledge
• Explicit: Information,
Explanation, Examples, Demos
• Implicit: Practice tasks/activities
(prompts and response)
• Diagnosis and feedback
• Explicit/Declarative/Conceptual/What
• Implicit/Procedural/How
• Knowledge Components
(Procedures + Facts, Concepts,
Principles, Processes)
• Response accuracy/errors
• Response fluency/speed
• Number of trials
• Amount of assistance (hints)
• Reasoning
Koedinger, K.R., Corbett, A.T., and Perfetti, C. (2010). The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning (Draft manuscript from the Pittsburgh Science of Learning Center)
13
5 types of outcomes determine TYPE of information and
practice
Knowledge
Component Definition Example
Procedure
Support
ive/C
onceptu
al Fact
Concept
Process
Principle
14
5 types of outcomes determine TYPE of information and
practice
Knowledge
Component Definition Example
Procedure
Sequence of decision and action
steps to perform tasks; when
and how to do things
•Prosecuting a criminal
•Deciding if capital gains
tax applies
Support
ive/C
onceptu
al Fact
Concept
Process
Principle
15
5 types of outcomes determine TYPE of information and
practice
Knowledge
Component Definition Example
Procedure
Sequence of decision and action
steps to perform tasks; when
and how to do things
•Prosecuting a criminal
•Deciding if capital gains
tax applies
Support
ive/C
onceptu
al Fact
Isolated, unique piece of
information; one instance
•52 Grosvenor Place
•2+3=5
Concept
Process
Principle
16
5 types of outcomes determine TYPE of information and
practice
Knowledge
Component Definition Example
Procedure
Sequence of decision and action
steps to perform tasks; when
and how to do things
•Prosecuting a criminal
•Deciding if capital gains
tax applies
Support
ive/C
onceptu
al Fact
Isolated, unique piece of
information; one instance
•52 Grosvenor Place
•2+3=5
Concept
Sets of items that share
common attributes, common
name; multiple examples
•Dog
•Money
•Happiness
Process
Principle
17
5 types of outcomes determine TYPE of information and
practice
Knowledge
Component Definition Example
Procedure
Sequence of decision and action
steps to perform tasks; when
and how to do things
•Prosecuting a criminal
•Deciding if capital gains
tax applies
Support
ive/C
onceptu
al Fact
Isolated, unique piece of
information; one instance
•52 Grosvenor Place
•2+3=5
Concept
Sets of items that share
common attributes, common
name; multiple examples
•Dog
•Money
•Happiness
Process Flow of events or procedures;
how things work
•Workflow
•Chemical process
Principle
18
5 types of outcomes determine TYPE of information and
practice
Knowledge
Component Definition Example
Procedure
Sequence of decision and action
steps to perform tasks; when
and how to do things
•Prosecuting a criminal
•Deciding if capital gains
tax applies
Support
ive/C
onceptu
al Fact
Isolated, unique piece of
information; one instance
•52 Grosvenor Place
•2+3=5
Concept
Sets of items that share
common attributes, common
name; multiple examples
•Dog
•Money
•Happiness
Process Flow of events or procedures;
how things work
•Workflow
•Chemical process
Principle
Guidelines, rules that govern,
predict, explain events;
relationships among concepts
•Supply and demand
•80/20 principle
•Novices need structure
19
3 stages of learning determine instructional elements
and sequence
Stage Characteristics Implications for
Instructional Design
1.
2.
3.
Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition.
Fitts & Posner, (1967), John Anderson (2004, 2007);
Anders Ericsson (2006, 2007)
20
3 stages of learning determine instructional elements
and sequence
Stage Characteristics Implications for
Instructional Design
1. Declarative
•Knowledge “about”, “that”,
“what” “why”;
•Can be stated verbally;
•Conceptual network
•Conscious
Design clear, relevant, and
accurate information displays, job
aids, examples, reference material
for all knowledge components:
facts, concepts, principles,
processes, procedures
2.
3.
Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition.
Fitts & Posner, (1967), John Anderson (2004, 2007);
Anders Ericsson (2006, 2007)
21
3 stages of learning determine instructional elements
and sequence
Stage Characteristics Implications for
Instructional Design
1. Declarative
•Knowledge “about”, “that”,
“what” “why”;
•Can be stated verbally;
•Conceptual network
•Conscious
Design clear, relevant, and
accurate information displays, job
aids, examples, reference material
for all knowledge components:
facts, concepts, principles,
processes, procedures
2. Procedural
•Knowledge “how”
•Sequence of “if-thens”
•Potential to become
unconscious
Design practice tasks to elicit
student performance/ responses;
monitoring systems to detect
errors; and feedback/coaching to
correct errors in performance
3.
Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition.
Fitts & Posner, (1967), John Anderson (2004, 2007);
Anders Ericsson (2006, 2007)
22
3 stages of learning determine instructional elements
and sequence
Stage Characteristics Implications for
Instructional Design
1. Declarative
•Knowledge “about”, “that”,
“what” “why”;
•Can be stated verbally;
•Conceptual network
•Conscious
Design clear, relevant, and
accurate information displays, job
aids, examples, reference material
for all knowledge components:
facts, concepts, principles,
processes, procedures
2. Procedural
•Knowledge “how”
•Sequence of “if-thens”
•Potential to become
unconscious
Design practice tasks to elicit
student performance/ responses;
monitoring systems to detect
errors; and feedback/coaching to
correct errors in performance
3. Automated
•Fluency
•Expert
•Unconscious
• “10,000 hours”
Design opportunities for repeated
frequent practice on the job and
monitoring of speed and accuracy
Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition.
Fitts & Posner, (1967), John Anderson (2004, 2007);
Anders Ericsson (2006, 2007)
23
Much research to guide us
Learning
Events (hidden - inside
students’ minds)
Student
Performance (observable -
indicates
knowledge)
Instructional
Events (in the learning
environment)
Knowledge
• Explicit: Information,
Explanation, Examples, Demos
• Implicit: Practice tasks/activities
(prompts and response)
• Diagnosis and feedback
• Explicit/Declarative/Conceptual/What
• Implicit/Procedural/How
• Knowledge Components
(Procedures + Facts, Concepts,
Principles, Processes)
• Response accuracy/errors
• Response fluency/speed
• Number of trials
• Amount of assistance (hints)
• Reasoning
Motivation
• Orientation/Inoculation
• Monitoring
• Diagnosis and treatment:
Persuasion, Modeling,
Dissonance
• Value beliefs
• Self-efficacy beliefs
• Attribution beliefs
• Mood/Emotion
• Behavior related to
• Starting
• Persisting
• Mental Effort
• Self-reported beliefs
Koedinger, K.R., Corbett, A.T., and Perfetti, C. (2010). The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning (Draft manuscript from the Pittsburgh Science of Learning Center)
24
4 beliefs influence motivation
Sources: Bandura; Eccles & Wigfield; Pintrich & Schunk; Clark; Dweck
Beliefs
• Value
• Self-Efficacy
• Attribution
• Mood
Motivated Behavior
• Starting
• Persisting
• Mental Effort
Learning/ Performance
• Practice
• Test
Self-Efficacy
Eff
ort
High Moderate Low
Motivation
Low High
Performance High
Low
• Design materials and interaction to foster positive mood, high
perception of value, moderate confidence, and attribution of
success and failure to effort
• Design system for monitoring and guidance (group and
individual)
25
Much research to guide us
Learning
Events (hidden - inside
students’ minds)
Student
Performance (observable -
indicates
knowledge)
Instructional
Events (in the learning
environment)
Knowledge
• Explicit: Information,
Explanation, Examples, Demos
• Implicit: Practice tasks/activities
(prompts and response)
• Diagnosis and feedback
• Explicit/Declarative/Conceptual/What
• Implicit/Procedural/How
• Knowledge Components
(Procedures + Facts, Concepts,
Principles, Processes)
• Response accuracy/errors
• Response fluency/speed
• Number of trials
• Amount of assistance (hints)
• Reasoning
Motivation
• Orientation/Inoculation
• Monitoring
• Diagnosis and treatment:
Persuasion, Modeling,
Dissonance
• Value beliefs
• Self-efficacy beliefs
• Attribution beliefs
• Mood/Emotion
• Behavior related to
• Starting
• Persisting
• Mental Effort
• Self-reported beliefs
Metacognition
• Structure
• Guidance
• Planning, Monitoring
• Selecting, Connecting
• Amount of guidance
required/requested
Koedinger, K.R., Corbett, A.T., and Perfetti, C. (2010). The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning (Draft manuscript from the Pittsburgh Science of Learning Center)
26
Agenda
• What evidence says about learning
• What this means for the design of instruction
• What happens when you do this for real
• [How to get outcomes aligned with real expertise]
• [“Teaching & Learning in the 21st Century” – thoughts]
27
Instructional design: “Engineering” from learning science
Overviews Information Examples Practice Assessment Learning
Outcomes
Motivational Guidance
Design
Deliver
Learning science strongly suggests an order to design and delivery
Clark, R.E., & Feldon, D. F. (2008). GEL (Guided Experiential Learning), Adaptable Expertise and Transfer of Training.
Kirscher, P.A., Sweller, J., & Clark, R. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of
constructivist, discovery, problem-based, experiential and inquiry-based teaching. Educational Psychologist, 41, 75-86.
Knowledge Integration
28
Evidence-based instructional principles
Accumulation of results from lab studies support: • Structure and guidance for novices
(Kirschner, Sweller, & Clark, 2006)
• Demonstrations and worked examples
(Paas & van Merrienboer, 1994; Sweller, 2006)
• Practice and corrective feedback
(Mathan & Koedinger, 2005)
• Prompted self-explanation
(Aleven & Koedinger, 2002)
• Multimedia use that minimizes extraneous cognitive load
(Mayer, 2009)
• Targeting beliefs (value, confidence, and attributions) and
emotions (positive feelings) to influence motivation
(Clark, 2004; Um et al., 2011)
29
Task-centered instruction
• Move from simple to increasingly difficult tasks – NOT “PBL” sink or swim
• Teach everything needed for each task
• Fade coaching/support over time
30
Knowledge
Component
Presentation (Prepare) Practice/Assessment (Practice, Perform)
Info Example Remember Proxy for Remember Use** Proxy for Use **
Procedure
When to use;
List of action and
decision steps
Demonstration of
when and how to
perform
Recall when
to use; Recall
action and
decision steps
Reorder steps;
Recall next or missing
steps
Decide when to use;
Perform the steps (actions
and decisions)
Critique performance
or output of actions
and decisions
Su
pp
ort
ive
Kn
ow
led
ge
Fact * Statement of
fact Statement of fact Recall fact
Recognize fact when
presented with distractors Recall fact in task context
Concepts
List of
defining
attributes
Examples;
Non-examples
List defining
attributes
verbally or in
writing
Recognize defining
attributes when presented
with distractors
Classify, identify or
generate examples and
non-examples
Critique someone
else’s identification or
generation of
examples
Process/
System
List of phases,
events and
causes at each
phase
Examples;
simulations of
phases, events,
and causes
Recall
phases,
events, and
causes
Recognize phases,
events, and causes;
Recall missing phases,
events, and causes
Identify causes of faults in
a process;
Predict events in a
process
Critique someone
else’s description of
causes or prediction of
events in a process
Principle
(cause and
effect
relationship)
Statement of
cause and effect
relationship
Examples,
demonstration,
simulation of
cause and effect
relationship
Recall the
principle
Recognize the principle;
Recall missing elements
of the principle
Decide if principle applies;
Predict an effect;
Apply principle to solve a
problem, explain a
phenomenon or make a
decision
Critique someone
else’s application of
the principle to solve a
problem, explain a
phenomenon or make
a decision
Knowledge
Integration
Explain the interconnections among
conceptual knowledge components, or
the conceptual foundation of
procedures, or the procedural
implementation of conceptual
knowledge components
Opportunities (including instructions, templates, rubrics) to self-explain, discuss, present, describe or
select their reasoning about interconnections among knowledge components, for example the
principle(s) that justify the application of a procedure.
Knowledge
Transfer
Multiple and varied contexts for
examples
Multiple and varied contexts for practice and assessment.
Opportunities for students to explain how they would use the knowledge in other contexts
*Facts are concepts with single instances
** All Use and Proxy for Use Activities develop/require procedural knowledge
Presentation and practice match objectives (knowledge components)
31
Agenda
• What evidence says about learning
• What this means for the design of instruction
• What happens when you do this for real
• [How to get outcomes aligned with real expertise]
• [“Teaching & Learning in the 21st Century” – thoughts]
32
ID can change instructional outcomes at scale
Principle Description Effect size
(s.d. units)
Multimedia Use relevant graphics and text to communicate content 1.5
Contiguity Integrate the text nearby the graphics on the screen – avoid covering or separating
integrated information
1.1
Coherence Avoid irrelevant graphics, stories, videos, media, and lengthy text 1.3
Modality Include audio narration where possible to explain graphic presentation 1.0
Redundancy Do not present words as both on-screen text and narration when graphics are
present
.7
Personalization Script audio in a conversational style using first and second person 1.3
Segmenting Break content down into small topic chunks that can be accessed t the learner’s
preferred rate
1.0
Pre-training Teach important concepts and facts prior to procedures or processes 1.3
Etc. Worked examples, self-explanation questions, varied-context examples and
comparisons, etc.
??
Source: E-learning and the Science of Instruction, Clark and Mayer, 2nd ed., 2008
34
Redeveloping courses at scale
Read, Write, Discuss
• Outcomes and content not
precisely aligned
• Limited demonstrations, worked
examples, and practice
• General assessment rubrics
• High reliance on discussion boards
Existing courses
Prepare, Practice, Perform
• Outcomes and content aligned
• One lesson per objective
• Demonstrations and worked examples
• Practice, feedback before assessment
• Detailed scoring guides
• Less discussion/more practice
• Standard instructor materials
• Monitoring and support for motivation
Redesigned courses
35
Content Design
Items Prepare Practice Perform
Seminar Discussion Lessons Sets
Overview
Course Level
Outcome 1
Unit
Outcome 1
Unit
Outcome 2
Unit
Outcome 3
Prepare
1
Practice
1
Perform
1
Prepare
2
Practice
2
Perform
2
Prepare
3
Practice
3
Perform
3
Lesson 1 Lesson 2 Lesson 3
Navigation
42
6
Overview
Seminar
Discussion
Journal: A situation in your life where the
guidelines for improving nonverbal communication
could guard against misinterpretation.
Lesson 1
Identify verbal and nonverbal elements in personal and professional
situations 1-2 hrs.
Lesson 2
Identify nonverbal communication principles in personal and professional situations 1-2 hrs.
Lesson 3
Explain instances of effective and ineffective communication in terms of
how verbal and nonverbal elements work together 2-4 hrs.
UNIT 5 Review: What is Nonverbal Communication?
2. Identify nonverbal communication principles in personal and professional situations
15
UNIT 5 Review: Lesson 2 Practice
http://www.youtube.com/watch?v=bg0kSIJZiRQ&feature=related
PART 1: Which nonverbal
communication principle is
predominant in the woman’s
reactions to her blind date?
Watch Item 2 video:
21
Overview (including Survey)
Seminar
Discussion
Explain how improving your listening skills
can increase the effectiveness of your
communication in the workplace and in your
personal life.
Lesson 1
Identify forms of nonlistening in personal and professional situations
1-2 hrs.
Lesson 2
Apply the principles of mindful listening to improve the effectiveness of communication in personal and professional situations 1-2 hrs.
UNIT 6 Preview: How Does Listening Enhance Our I.C.?
25
UNIT 6: Lesson 2 Practice
Watch the Online Dating video. Answer the three questions, referring to the scoring guide.
Online Dating
After answering a question, study the
“Compare with Expert” response.
1. From the interaction does it seem to you that Chris’s mom is actively listening during the first
third of this conversation? Why or why not?
2. Apply the principles of mindful listening to improve communication effectiveness
29
Agenda Minutes
Opening 5 P
Student Questions 10 P
Review Unit 5 10 P
View Unit 6 25 P
Preview Unit 7 5 P
Wrap Up 5 P
Poll Question
How many of you still
have questions?
Post your questions in
“Course Questions” discussion board
q Yes q No
(Link in Course Home menu)
Wrap Up
Instructor seminar materials more standardized and
aligned with online content
44
What happened?
Research design: quasi-experimental
Control Pilot Control Pilot Control Pilot
Interpersonal Communications 8 7 4 3 237 199
Principles of Nutrition 6 4 3 2 148 89
Medical Terminology 6 7 4 2 197 220
Total 20 18 11 7 582 508
n sections n instructors n studentsCourse
• 1,090 students (508 pilot; 582 control)
• 87% female, average age 32; average household income $20,000
• 3 courses
• 18 instructors
• 20 sections (assigned to pilot or control)
• 2 terms (Aug – Dec 2011)
45
Analysis
• Logistic regression to examine effect of course design
on student success
• Success (1 or 0): Defined as:
Pass (1 or 0)
+ Master course objectives (>=4 on 0-5 scale)
+ Stay (1 or 0)
• Controlled for variation in
• Instructor prior student success rates
• Student background variables • Age, prior education, prior GPA, tenure, household
income
• Calendar-based success variation
46
Student success: results controlling for variables
• 11% higher
success rate
• 28% increase
• Students in
redesigned
courses were
1.6 times
more likely to
be successful
Wald Chi-Square: 10.42, df=1, n=895, Sig<.001.
39%
50%
20%
30%
40%
50%
60%
70%
Control Pilot
Adju
ste
d s
tudent success r
ate
Adjusted student success rates with 95% confidence limits
47
More to do: adjusted results varied by course
• Odds of success more than doubled (2.5 times more) in Interpersonal
Communications
• 9% difference in Principles of Nutrition (not stat significant due to smaller sample)
• Small improvement in Medical Terminology due to difficulty level of early units
48
Student quote on benefits of added practice
“Something I found to be interesting was the degree of understanding between me and another individual that wasn’t in this class. A girl I had met in a previous term that has a similar degree plan but ended up in a regular medical terminology course, still we would discuss the differences and similarities between are assigned classes. During our unit 8 test she called me hysterical about all the different elements of the final tests and couldn’t seem to grasp the concept of the 1st part of the test i.e., analysis diagram, creating new terms from word roots etc. I was mystified that something that had become 2nd nature to me mainly due to the time spent every week filling out the Analysis Tables was so difficult for her to comprehend. It was at that point I realized all the griping I had done was actually the reason my level of understanding is more evolved than somebody who never experienced it.”
49
Quote from a student who previously failed
“This course was difficult for me to do. I
tried to do this course when I attended
another school and I failed it. I think the way
the course was set up and how it broke
everything down really helped me to
understand it and pass it this time. I would
not change a thing about how this course
was set up.”
50
Satisfaction (end of term survey, 5 point scale)
Student satisfaction • Lower on end of course survey in redesigned courses (mean 4.4
vs 4.8), but still greater than 4 on 5 point scale. High positivity
scores in motivation survey.
Why?
• Courses more rigorous, more work to complete; this is a
common finding in other research (e.g., Clark, 1982)
Instructor satisfaction • Higher in redesigned courses (mean 4.6 vs 4.1)
Why?
• Detailed scoring guides for assignments
• Less time in discussions – more time to monitor and
communicate with at-risk students
• Standard seminar format and content
• Student materials: structure, clarity, practice
51
Agenda
• What evidence says about learning
• What this means for the design of instruction
• What happens when you do this for real
• [How to get outcomes aligned with real expertise]
• [“Teaching & Learning in the 21st Century” – thoughts]
52
Employers actually expect job applicants to lack the
occupational/technical skills required to do the job…
• Slightly over half of all
respondents (52.8%)
expected that job applicants
would lack occupational
skills
• In healthcare, where
occupational certifications and
licensures are required, over
68% of respondents expect
that job applicants would lack
occupational skills
Do you expect job applicants to be lacking specific occupational
skills or technical skills?
March 2011 Workforce Connections, Inc. survey of employers in western Wisconsin. Over 400 employers
from all 8 counties responded to the survey. All sizes of businesses were represented with the majority of
responses coming from businesses with less than 50 employees.
53
… and end up investing significantly in training (if they
can afford it)
What area of training comprises the
bulk of your training budget?
• Of the companies that have
training budgets, 68% of the
budget is allocated towards
skills training for new workers
• “The cost of losing and replacing
an experienced paralegal is …
roughly $100,000.”1
• “Annual paralegal turnover is
nearly 50% a year in many large
firms”…. 2 (and)”… is about 28%
nationally”. 3
1 Source: American Bar Association Standing Committee on Paralegals
(2001)http://apps.americanbar.org/legalservices/paralegals/update/campbellarticle.html 2 Greene, A. and Cannon, T. (2003) Paralegals, profitability and the future of your law practice. 3 Jordan, P. D. (2001) Paralegal Studies (quoting from Bureau of Labor Statistics).
Poorly trained employees drive high
turnover rates
54
CTA lets us do better than letting experts teach
• Experts are mostly unable, unaided, to express in words to novices more
than 30% of their decision-making
- They can visualize procedures, but not cognitive decisions
• CTA gets to 70-80% of expert decision-making
- Structured interviews with objectively-determined experts
- Refined to a “gold standard” of decisions and tasks
• When coupled with well-structured training (see Kaplan Way), takes 20%+
less effort, increases student learning by 25%+ with fewer errors
• Can influence or change how professions think about themselves – but may
have to leave some “hoops” for training to gain acceptance
55
CTA methods can help learning environments work better
• CTA methods have evidence they unlock 40-50% more of experts’ skills, reduce
training time, and increase student’s confidence
• CTA provides the inputs (including task scenarios) to inform high-quality, complete,
task-centered instruction
From patent
examiner CTA:
56
• Commanding Generals
• Financial Analysts
• Trauma Surgeons
• Fire Chiefs
• Geologists
• Salespeople
• Pharmacological Researchers
• Experimental Psychologists
• Patent Examiners
• Research Librarians
• Nuclear Generator Design
Engineers
• Psychotherapists
• Chemists
• Radiological Cardiologists
• Neonatal Nurses
• Classroom Teachers
• Fighter Pilots
• SWAT Teams
• Emergency Room Teams
• Football Coaches
• Blackjack winners (21)
• Chicken Sexers
• . . .
A wide array of professions have already used CTAs
57
CTA has made real differences in training time and
learner success by investing in design up-front
Medical school surgical instruction
CTA-trained surgeons had greater gains
from pretest to post-test in less time
Also outperformed control group on patients
in every measure of performance
Kaplan University Online Faculty
CTA-based assessment instrument identified
faculty whose students achieved 5% less
Urate and .5 higher GPA on average than
others teaching the same courses.
KU is currently developing hiring tools and
training to take advantage of the strategies.
Spreadsheet training
Scores on post-test problems, and average
time to completion:
• Discovery learning: 34% - 60 minutes
• Guided demonstration: 64% - 49 minutes
• CTA: 89% - 29 minutes
Emergency and safety procedures
New course took half the time with higher
scores on the performance posttest
CTA required 85% more front-end time for
design, development, and PD
58
CTA is a systematic way to document expertise
• CTA is an interview strategy for capturing how highly successful experts perform complex tasks in a variety of settings
• Goal is to develop authentic demonstration and practice opportunities for how to perform at expert levels
• Experts are interviewed who 1) have recent (past 2-3 mo.) experience, 2) are consistently successful, and 3) are NOT trainers.
• Interviews are done with 3-4 experts to unpack their strategies; these are merged to make an efficient approach suitable for training
• A range of problem examples or performance scenarios are collected from the experts for use in instruction as well
59
Medical Assistant current course content:
X = substantial content; x = ancillary content
Pharmacology course
Diseases - human body
60
MA CTA: Identifies key tasks/skills performed by experts
Tool Skills
Clinical Skills
Patient Assessment
First Aid
Medical History
Patient exam prep
Administering Medication
Remove sutures
Vital signs
Patient Education
Prep and clean exam room
Collect samples and lab specimens
Administrative Skills
Medical records
Prior authorization
Communication
Computer use
Ethical skills
Supply maintenance
Laboratory Skills
Preparation of
medication
Preparation of
samples and lab
specimens
Course Content
Medical Law
and Bioethics
Medical Terminology
Anatomy and
Physiology
Pharmacology
Diseases of the Human Body
Medical Office
Management
Medical Coding and Insurance
Professionalism
in Health Care
Clinical Competencies
Original content New focus
• Tie to domain tasks as
identified by experts
61
MA Program: Skills addressed in new sequence
Tool Skills
Clinical Skills
Patient Assessment
First Aid
Medical History
Patient exam prep
Administering Medication
Remove sutures
Vital signs
Patient Education
Prep and clean exam room
Collect samples and lab specimens
Administrative Skills
Medical records
Prior authorization
Communication
Computer use
Ethical skills
Supply maintenance
Laboratory Skills
Preparation of
medication
Preparation of
samples and lab
specimens
New focus
• Tie to domain tasks as
identified by experts
62
Task-centered instruction
• Move from simple to increasingly difficult tasks – NOT “PBL” sink or swim
• Teach everything needed for each task
• Fade coaching/support over time
63
MA Program: Skills addressed in new sequence
• Tie to domain tasks as
identified by experts
• Repeated use of skills
across courses
B: Begin; A: Advanced; R: Reinforce
64
MA Program: New courses include previous content
Original Course Content
Proposed Course Sequence
Admin Skills 1
Clinical Skills 1
Admin Skills 2
Clinical Skills 2
Lab Skills
Medical Law and Bioethics
X x X x x
Medical Terminology
X X X X X
Anatomy and
Physiology X X X
Pharmacology X X X
Diseases of the
Human Body X X x
Medical Office Management
X X X x X
Medical Coding and Insurance
X X X
Professionalism
in Health Care X X x x X
Clinical Competencies
X X X X X
X = substantial content; x = ancillary content
• Tie to domain tasks as
identified by experts
• Repeated use of skills
across courses
• Original concepts spread
across task instruction,
not confined to courses
65
Agenda
• What evidence says about learning
• What this means for the design of instruction
• What happens when you do this for real
• [How to get outcomes aligned with real expertise]
• [“Teaching & Learning in the 21st Century” – thoughts]
67
Diversity?
• Absolutely
• An expectation that a wider variance of already-
mastered skills has to be accommodated
• Much more flexibility around the logistics of mastery –
schedules, life-stage, life changes, etc.
• New work to help the diverse array of learners really
understand what works for learning (Carol Dweck)
• Arguably, similar issues for faculty – learners too!
68
More innovative pedagogies and experiential learning?
• Yes – but need to be taking into account how our
learning machinery actually works (medical analogy)
-Whether you do or don’t – the real world awaits. . .
• Guided/structured experiential learning for novices
• Closer tie between each concept and applications– both
must get taught and practiced in close proximity
• Challenge: How to balance individual instructor
innovation with lessons learned from thousands?
69
More collaborative?
• Collaborating to solve a hard problem at work is not the
same as collaborating to learn
• Learning to collaborate as one does at work is a terrific
goal – but is just as hard as domain-specific objectives
• For novices, collaboration to learn too soon can be too
much cognitive load – may block mastery of key
objectives
• Group activities after mastery are terrific for cementing,
extending, generalizing
70
Supported by technology?
• Technology does NOT solve learning problems per se
(or any other problem)
• Technology takes a good (or bad) solution, and makes it
more affordable, reliable, available, data-rich, etc.
• So it is and will be a critical component of educational
systems – much more than now
• But if learning does not take precedence. . .
• A key is much more systematic use of data – to evaluate
measures of learning, processes for learning, to
personalize learning to student’s needs
• Technology (well-deployed) should help us find what
works and deploy it systematically and well
71
Changing student expectations?
• Students should have the right to expect that what they learn is deeply
tied to what experts really decide and do – variable now, most careers!
• Student’s expectations of a “good” environment are not always correct
– especially for novices in a domain
• Indeed, research shows students often think environments that work
better for them work worse, and vice-versa (Steve-Jobs-like lesson!)
• Culture can get in the way, too – beliefs about talent/learning
• Students brains are NOT “rewired now”
- It’s the same machinery (narrow working memory supported by fast
long term memory)
-There are new things driven into long-term memory, however
-The system works as it has – neurons don’t follow Moore’s law!
-E.g.: multi-tasking – can produce, but not become better
72
A tool we’re using: An evidence-based checklist –
specifications for design and quality assurance
Is the course/lesson designed for effective
knowledge acquisition and transfer?
• Learning outcomes/objectives
• Assessments
• Practice
• Presentation: Examples
• Presentation: Information
• Content chunking and sequencing
Does the course provide support for
motivation?
Does the course provide opportunities for
knowledge integration?
Are media used appropriately and efficiently?
Does instruction adapt to student's level of
knowledge and motivation?
The checklist Categories on the checklist
73
Where do items on checklist come from?
Principle Design Actions
Task-
centeredness
Include authentic
tasks that represent
the domain/learning
outcomes
Activation
Connect to learner’s
prior
experience/knowledge
/larger knowledge
structure
Demonstration
Demonstrate and give
examples of correct
performance
Application
Provide part-task and
whole-task practice
with corrective
feedback
Integration
Deepen knowledge
with opportunities for
reflection, discussion,
public performance,
exploration of real life
uses
Merrill, M. D. “First Principles of Instruction,” In C. M.
Reigeluth & A. Carr (Eds.), Instructional Design
Theories and Models III (Vol. III), 2009
First Principles of Instruction
Principle Design Actions
Multimedia Use words and graphics rather than words alone
Contiguity Place printed words near corresponding graphics; Synchronize spoken words
with corresponding graphics
Modality Present words as audio narration rather than on-screen text
Redundancy Explain visuals with words in audio OR text, not both
Coherence Avoid interesting but unnecessary material; avoid extraneous audio,
graphics, words
Personalization Use conversational rather than formal style; Use effective on-screen
coaches; Make the author visible
Segmenting Break content into bite-size segments
Pre-training Teach key concepts prior to procedures or processes
Examples
Transition from worked examples to problems via fading; Promote self-
explanation of worked-out steps; Supplement worked examples with
explanations
Practice
Mirror the job; Provide explanatory feedback; Adapt the amount and
placement of practice to job performance requirements; Transition from
examples to practice gradually
Collaboration Insufficient evidence for guidelines on social learning
Learner Control/
Navigation
Give experienced learners control; Make important instructional events the
default; Consider adaptive control; Give pacing control
Build Thinking
Skills
Use job-specific cases; Make thinking processes explicit; Define job-specific
problem-solving processes
Games and
Simulations
Match game type to learning goals; Make learning essential to progress;
Build in guidance; Promote Reflection on correct answers; Manage
complexity
Source: E-learning and the Science of
Instruction, Clark and Mayer, 2nd ed., 2008
E-Learning and Multimedia Design Principles
74
Evidence-based checklist:
Objectives, Assessment, Practice
1 Learning Outcomes/Objectives
1.1 Learning objectives are stated. 0.0
1.2 Learning objectives are stated as performance objectives, i .e., what learners will be able to DO, not what they will know. 0.0
1.3 Lesson, module, units, course, and program objectives are aligned. 0.0
1.4Learning objectives map to certification requirements or competencies or domain taxonomies/standards from professional or
accreditation bodies. 0.0
1.5 Learning objectives are based on cognitive task analysis of expert performance in the domain or profession. 0.0
SECTION SCORE 0.0
2 Assessment
2.1 Assessment tasks match learning outcomes/objectives. 0.0
2.2Assessment tasks measure mastery/acquisition of knowledge components: procedures, facts, concepts, principles, processes (one
assessment may cover multiple objectives). 0.0
2.3 Rubrics (scoring guides) guide scoring and performance of assessment tasks with open-ended response formats. 0.0
SECTION SCORE 0.0
3 Practice
3.1 Practice matches assessment. 0.0
3.2Practice tasks elicit performance to develop procedural knowledge and supportive knowledge components (facts, concepts, principles,
processes). 0.0
3.3 Rules/rubrics diagnose errors and misconceptions. 0.0
3.4 Feedback/adaptation/guidance corrects errors and misconceptions. 0.0
3.5 Practice matches transfer context, e.g., job situation. (Prompt is contextually authentic. Response is cognitively authentic). 0.0
3.6 Part-task practice precedes whole task practice. 0.0
SECTION SCORE 0.0
75
Presentation: Examples and Information
4 Presentation: Examples
4.1 Examples (demonstrations, worked examples) match practice. 0.0
4.2 Demonstrations (or worked examples) i l lustrate task performance (procedures). 0.0
4.3 Examples, stories, cases il lustrate concepts, principles, processes. 0.0
SECTION SCORE 0.0
5 Presentation: Information
5.1Descriptions and explanations cover steps to perform a task (when and how), and related knowledge components - facts, concepts,
principles, processes (what and why). 0.0
5.2 Information needed to do practice tasks is emphasized; "nice to know" information is excluded or minimized. 0.0
5.3 Information is integrated (interwoven) with examples. 0.0
SECTION SCORE 0.0
76
Chunking, Sequencing, Overviews
Do course, unit, and lesson overviews support learning and motivation?
9 Overviews
9.1Overviews include orientation (description of where each component - course, unit, lesson - fits in a larger curriculum, program, course,
process, or hierarchy of objectives). 0.0
9.2 Goals are clarified (description of the learning outcomes or objectives - what learner will be able to do by end). 0.0
9.3 Value/Reasons/Benefits/Risks are explained (inoculation against low perceived value of content and/or methods). 0.0
9.4 Connection to prior knowledge/something familiar is made or activated (story, example, analogy, questions). 0.0
9.5 Outlines describe what is to come (in the course, unit, or lesson). 0.0
SECTION SCORE 0.0
6 Content Chunking and Sequencing
6.1 Content is broken into manageable chunks/segments. 0.0
6.2 Outcomes/objectives are presented in order of application, difficulty, with prerequisites first. 0.0
6.3 For each outcome/objective, the learning sequence is Overview, Presentation (Information and Examples), Practice, Assessment. 0.0
6.4 Navigation from section to section is simple and not confusing - there is a clear sequence and clear directions. 0.0
SECTION SCORE 0.0
77
Multimedia
Are media used appropriately and efficiently?
10 Multimedia
10.1 Graphics and media are relevant and not distracting (Coherence Principle). 0.0
10.2 Text and graphics are positioned close together without scrolling (Contiguity Principle). 0.0
10.3 Visuals are explained with text or audio, not both (Redundancy Principle). 0.0
10.4 For complex graphics, audio narration is used instead of on-screen text (Modality Principle). 0.0
10.5 Students control pace of media to play, pause, forward, rewind (Pacing Control Principle). 0.0
10.6Media use is consistent with Section 508 of the Americans with Disabilities Act (e.g., non-text has text equivalent; images are tagged with
text, audio has text alternative). 0.0
10.7 Look and feel are polished; media quality is adequate. 0.0
SECTION SCORE 0.0
78
Motivation, Knowledge Integration, Personalization Does the course provide support for motivation?
7 Motivation
7.1 References are made to importance/reasons for the content and/or instructional methods used. 0.0
7.2 The tone is positive, encouraging, conversational. 0.0
7.3 Statements or stories are included to attribute success and failure to effort, not innate ability. 0.0
7.4 Statements or stories are included to prevent under-confidence/anxiety and over-confidence. 0.0
SECTION SCORE 0.0
Does the course provide opportunities for knowledge integration?
8 Integration
8.1Techniques to promote deep processing and integration of knowledge are included, for example, prompted self-reflection, self-explanation,
discussions, student presentations. 0.0
SECTION SCORE 0.0
79
References:
• Why Students Don’t Like School, Daniel Willingham – highly readable! ;-)
• Talent is Overrated, Geoffrey Colvin – highly readable! ;-)
• E-Learning and the Science of Instruction, Clark and Mayer, 2nd ed.
• “First Principles of Learning,” Merrill, D., in Reigeluth, C. M. & Carr, A. (Eds.),
Instructional Design Theories and Models III, 2009.
• How People Learn, John Bransford et al, eds.
• “Design factors for educationally effective animations and simulations,” Plass,
J.L., Homer, B.D., Hayward, E.O., J Comput High Educ (2009) 21:31–61
• “The Implications of Research on Expertise for Curriculum and Pedagogy”,
David Feldon, Education Psychology Review (2007) 19:91–110
• “Cognitive Task Analysis,” Clark, R.E., Feldon, D., van Merrienboer, J., Yates,
K., and Early, S.. in Spector, J.M., Merrill, M.D., van Merrienboer, J. J. G., &
Driscoll, M. P. (Eds.), Handbook of research on educational
communciatinos and technology (3rd ed., 2007) Lawrence Erlbaum
Associates