evidence-based thinking about learning and...

79
Evidence-based thinking about learning and instruction University of Washington May 3, 2012

Upload: lamquynh

Post on 06-Jul-2018

220 views

Category:

Documents


0 download

TRANSCRIPT

Evidence-based thinking about

learning and instruction

University of Washington

May 3, 2012

2

• Integrating the design, building, monitoring, and improvement of learning

environments; individualize learning experiences using our scale; and,

ultimately, drive greater student career success.

• Former CLO for K12, Inc. – structured use of technology, cognitive

science, on-line and off-line materials for 1,700 teachers, 55k students

• Former Publisher and General Manager for DK Multimedia, Inc.

• Management consultant with McKinsey & Company

• Education:

- Ph.D. in Electrical Engineering and Computer Science from MIT

- M.D. from Harvard Medical School

- M.A. in Electrical Engineering and Computer Science from MIT

- M.A. in Mathematics from Oxford University

- B.S. in Electrical Engineering and B.S. with Honors in Mathematics

from the University of Washington

Bror Saxberg Chief Learning Officer, Kaplan, Inc.

3

• Kaplan University

• Kaplan Legal Education

• Kaplan Professional

Education

• Nursing

• Kaplan Continuing

Education

• KNEXT

• KTPA

• Kaplan Tutoring

• Kaplan Bar Review

• Kaplan Publishing

• Kaplan Higher Ed – Europe

• Kaplan Professional – Europe

• Kaplan Higher Ed – Asia

• Kaplan Professional – Asia

• Kaplan Higher Ed – Australia

• Kaplan Professional –

Australia

• In Country Pathways – China

• Franklyn Scholar

• Carrick Education

• Global Knowledge Solutions

Kaplan education spans domains and geography

Kaplan University

Group

Kaplan Higher

Education Campuses

Kaplan Test Prep

Kaplan Asia Pacific Kaplan United

Kingdom • Kaplan Int’l Colleges

• Global Pathways

Kaplan International Colleges

4

What Our Students Told Us They Want from a World’s Best Educator

Promise

Pillars

Definitions

We strive to make

education as

personalized to you

as possible−tailoring

our courses around

your individual

needs.

We are dedicated

to getting you the

results that matter

in the time that

matters.

We move quickly

with constant

innovation to

better meet your

needs.

We are here to

help you achieve

success at critical

milestones along

your educational

journey.

5

What we’re trying to do

6

What we’re trying to do

7

What we’re trying to do

8

What we’re trying to do

9

What we’re trying to do

10

Agenda

• What evidence says about learning

• What this means for the design of instruction

• What happens when you do this for real

• [How to get outcomes aligned with real expertise]

• [“Teaching & Learning in the 21st Century” – thoughts]

11

Agenda

• What evidence says about learning

• What this means for the design of instruction

• What happens when you do this for real

• [How to get outcomes aligned with real expertise]

• [“Teaching & Learning in the 21st Century” – thoughts]

12

Much research to guide us

Learning

Events (hidden - inside

students’ minds)

Student

Performance (observable -

indicates

knowledge)

Instructional

Events (in the learning

environment)

Knowledge

• Explicit: Information,

Explanation, Examples, Demos

• Implicit: Practice tasks/activities

(prompts and response)

• Diagnosis and feedback

• Explicit/Declarative/Conceptual/What

• Implicit/Procedural/How

• Knowledge Components

(Procedures + Facts, Concepts,

Principles, Processes)

• Response accuracy/errors

• Response fluency/speed

• Number of trials

• Amount of assistance (hints)

• Reasoning

Koedinger, K.R., Corbett, A.T., and Perfetti, C. (2010). The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning (Draft manuscript from the Pittsburgh Science of Learning Center)

13

5 types of outcomes determine TYPE of information and

practice

Knowledge

Component Definition Example

Procedure

Support

ive/C

onceptu

al Fact

Concept

Process

Principle

14

5 types of outcomes determine TYPE of information and

practice

Knowledge

Component Definition Example

Procedure

Sequence of decision and action

steps to perform tasks; when

and how to do things

•Prosecuting a criminal

•Deciding if capital gains

tax applies

Support

ive/C

onceptu

al Fact

Concept

Process

Principle

15

5 types of outcomes determine TYPE of information and

practice

Knowledge

Component Definition Example

Procedure

Sequence of decision and action

steps to perform tasks; when

and how to do things

•Prosecuting a criminal

•Deciding if capital gains

tax applies

Support

ive/C

onceptu

al Fact

Isolated, unique piece of

information; one instance

•52 Grosvenor Place

•2+3=5

Concept

Process

Principle

16

5 types of outcomes determine TYPE of information and

practice

Knowledge

Component Definition Example

Procedure

Sequence of decision and action

steps to perform tasks; when

and how to do things

•Prosecuting a criminal

•Deciding if capital gains

tax applies

Support

ive/C

onceptu

al Fact

Isolated, unique piece of

information; one instance

•52 Grosvenor Place

•2+3=5

Concept

Sets of items that share

common attributes, common

name; multiple examples

•Dog

•Money

•Happiness

Process

Principle

17

5 types of outcomes determine TYPE of information and

practice

Knowledge

Component Definition Example

Procedure

Sequence of decision and action

steps to perform tasks; when

and how to do things

•Prosecuting a criminal

•Deciding if capital gains

tax applies

Support

ive/C

onceptu

al Fact

Isolated, unique piece of

information; one instance

•52 Grosvenor Place

•2+3=5

Concept

Sets of items that share

common attributes, common

name; multiple examples

•Dog

•Money

•Happiness

Process Flow of events or procedures;

how things work

•Workflow

•Chemical process

Principle

18

5 types of outcomes determine TYPE of information and

practice

Knowledge

Component Definition Example

Procedure

Sequence of decision and action

steps to perform tasks; when

and how to do things

•Prosecuting a criminal

•Deciding if capital gains

tax applies

Support

ive/C

onceptu

al Fact

Isolated, unique piece of

information; one instance

•52 Grosvenor Place

•2+3=5

Concept

Sets of items that share

common attributes, common

name; multiple examples

•Dog

•Money

•Happiness

Process Flow of events or procedures;

how things work

•Workflow

•Chemical process

Principle

Guidelines, rules that govern,

predict, explain events;

relationships among concepts

•Supply and demand

•80/20 principle

•Novices need structure

19

3 stages of learning determine instructional elements

and sequence

Stage Characteristics Implications for

Instructional Design

1.

2.

3.

Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition.

Fitts & Posner, (1967), John Anderson (2004, 2007);

Anders Ericsson (2006, 2007)

20

3 stages of learning determine instructional elements

and sequence

Stage Characteristics Implications for

Instructional Design

1. Declarative

•Knowledge “about”, “that”,

“what” “why”;

•Can be stated verbally;

•Conceptual network

•Conscious

Design clear, relevant, and

accurate information displays, job

aids, examples, reference material

for all knowledge components:

facts, concepts, principles,

processes, procedures

2.

3.

Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition.

Fitts & Posner, (1967), John Anderson (2004, 2007);

Anders Ericsson (2006, 2007)

21

3 stages of learning determine instructional elements

and sequence

Stage Characteristics Implications for

Instructional Design

1. Declarative

•Knowledge “about”, “that”,

“what” “why”;

•Can be stated verbally;

•Conceptual network

•Conscious

Design clear, relevant, and

accurate information displays, job

aids, examples, reference material

for all knowledge components:

facts, concepts, principles,

processes, procedures

2. Procedural

•Knowledge “how”

•Sequence of “if-thens”

•Potential to become

unconscious

Design practice tasks to elicit

student performance/ responses;

monitoring systems to detect

errors; and feedback/coaching to

correct errors in performance

3.

Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition.

Fitts & Posner, (1967), John Anderson (2004, 2007);

Anders Ericsson (2006, 2007)

22

3 stages of learning determine instructional elements

and sequence

Stage Characteristics Implications for

Instructional Design

1. Declarative

•Knowledge “about”, “that”,

“what” “why”;

•Can be stated verbally;

•Conceptual network

•Conscious

Design clear, relevant, and

accurate information displays, job

aids, examples, reference material

for all knowledge components:

facts, concepts, principles,

processes, procedures

2. Procedural

•Knowledge “how”

•Sequence of “if-thens”

•Potential to become

unconscious

Design practice tasks to elicit

student performance/ responses;

monitoring systems to detect

errors; and feedback/coaching to

correct errors in performance

3. Automated

•Fluency

•Expert

•Unconscious

• “10,000 hours”

Design opportunities for repeated

frequent practice on the job and

monitoring of speed and accuracy

Anderson, J.R. (1993). Rules of the Mind. Mahwah, NJ, Lawrence Erlbaum. Erricsson, A. & Charness,, Expertise: Its Structure and Acquisition.

Fitts & Posner, (1967), John Anderson (2004, 2007);

Anders Ericsson (2006, 2007)

23

Much research to guide us

Learning

Events (hidden - inside

students’ minds)

Student

Performance (observable -

indicates

knowledge)

Instructional

Events (in the learning

environment)

Knowledge

• Explicit: Information,

Explanation, Examples, Demos

• Implicit: Practice tasks/activities

(prompts and response)

• Diagnosis and feedback

• Explicit/Declarative/Conceptual/What

• Implicit/Procedural/How

• Knowledge Components

(Procedures + Facts, Concepts,

Principles, Processes)

• Response accuracy/errors

• Response fluency/speed

• Number of trials

• Amount of assistance (hints)

• Reasoning

Motivation

• Orientation/Inoculation

• Monitoring

• Diagnosis and treatment:

Persuasion, Modeling,

Dissonance

• Value beliefs

• Self-efficacy beliefs

• Attribution beliefs

• Mood/Emotion

• Behavior related to

• Starting

• Persisting

• Mental Effort

• Self-reported beliefs

Koedinger, K.R., Corbett, A.T., and Perfetti, C. (2010). The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning (Draft manuscript from the Pittsburgh Science of Learning Center)

24

4 beliefs influence motivation

Sources: Bandura; Eccles & Wigfield; Pintrich & Schunk; Clark; Dweck

Beliefs

• Value

• Self-Efficacy

• Attribution

• Mood

Motivated Behavior

• Starting

• Persisting

• Mental Effort

Learning/ Performance

• Practice

• Test

Self-Efficacy

Eff

ort

High Moderate Low

Motivation

Low High

Performance High

Low

• Design materials and interaction to foster positive mood, high

perception of value, moderate confidence, and attribution of

success and failure to effort

• Design system for monitoring and guidance (group and

individual)

25

Much research to guide us

Learning

Events (hidden - inside

students’ minds)

Student

Performance (observable -

indicates

knowledge)

Instructional

Events (in the learning

environment)

Knowledge

• Explicit: Information,

Explanation, Examples, Demos

• Implicit: Practice tasks/activities

(prompts and response)

• Diagnosis and feedback

• Explicit/Declarative/Conceptual/What

• Implicit/Procedural/How

• Knowledge Components

(Procedures + Facts, Concepts,

Principles, Processes)

• Response accuracy/errors

• Response fluency/speed

• Number of trials

• Amount of assistance (hints)

• Reasoning

Motivation

• Orientation/Inoculation

• Monitoring

• Diagnosis and treatment:

Persuasion, Modeling,

Dissonance

• Value beliefs

• Self-efficacy beliefs

• Attribution beliefs

• Mood/Emotion

• Behavior related to

• Starting

• Persisting

• Mental Effort

• Self-reported beliefs

Metacognition

• Structure

• Guidance

• Planning, Monitoring

• Selecting, Connecting

• Amount of guidance

required/requested

Koedinger, K.R., Corbett, A.T., and Perfetti, C. (2010). The Knowledge-Learning-Instruction (KLI) Framework: Toward Bridging the Science-Practice Chasm to Enhance Robust Student Learning (Draft manuscript from the Pittsburgh Science of Learning Center)

26

Agenda

• What evidence says about learning

• What this means for the design of instruction

• What happens when you do this for real

• [How to get outcomes aligned with real expertise]

• [“Teaching & Learning in the 21st Century” – thoughts]

27

Instructional design: “Engineering” from learning science

Overviews Information Examples Practice Assessment Learning

Outcomes

Motivational Guidance

Design

Deliver

Learning science strongly suggests an order to design and delivery

Clark, R.E., & Feldon, D. F. (2008). GEL (Guided Experiential Learning), Adaptable Expertise and Transfer of Training.

Kirscher, P.A., Sweller, J., & Clark, R. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of

constructivist, discovery, problem-based, experiential and inquiry-based teaching. Educational Psychologist, 41, 75-86.

Knowledge Integration

28

Evidence-based instructional principles

Accumulation of results from lab studies support: • Structure and guidance for novices

(Kirschner, Sweller, & Clark, 2006)

• Demonstrations and worked examples

(Paas & van Merrienboer, 1994; Sweller, 2006)

• Practice and corrective feedback

(Mathan & Koedinger, 2005)

• Prompted self-explanation

(Aleven & Koedinger, 2002)

• Multimedia use that minimizes extraneous cognitive load

(Mayer, 2009)

• Targeting beliefs (value, confidence, and attributions) and

emotions (positive feelings) to influence motivation

(Clark, 2004; Um et al., 2011)

29

Task-centered instruction

• Move from simple to increasingly difficult tasks – NOT “PBL” sink or swim

• Teach everything needed for each task

• Fade coaching/support over time

30

Knowledge

Component

Presentation (Prepare) Practice/Assessment (Practice, Perform)

Info Example Remember Proxy for Remember Use** Proxy for Use **

Procedure

When to use;

List of action and

decision steps

Demonstration of

when and how to

perform

Recall when

to use; Recall

action and

decision steps

Reorder steps;

Recall next or missing

steps

Decide when to use;

Perform the steps (actions

and decisions)

Critique performance

or output of actions

and decisions

Su

pp

ort

ive

Kn

ow

led

ge

Fact * Statement of

fact Statement of fact Recall fact

Recognize fact when

presented with distractors Recall fact in task context

Concepts

List of

defining

attributes

Examples;

Non-examples

List defining

attributes

verbally or in

writing

Recognize defining

attributes when presented

with distractors

Classify, identify or

generate examples and

non-examples

Critique someone

else’s identification or

generation of

examples

Process/

System

List of phases,

events and

causes at each

phase

Examples;

simulations of

phases, events,

and causes

Recall

phases,

events, and

causes

Recognize phases,

events, and causes;

Recall missing phases,

events, and causes

Identify causes of faults in

a process;

Predict events in a

process

Critique someone

else’s description of

causes or prediction of

events in a process

Principle

(cause and

effect

relationship)

Statement of

cause and effect

relationship

Examples,

demonstration,

simulation of

cause and effect

relationship

Recall the

principle

Recognize the principle;

Recall missing elements

of the principle

Decide if principle applies;

Predict an effect;

Apply principle to solve a

problem, explain a

phenomenon or make a

decision

Critique someone

else’s application of

the principle to solve a

problem, explain a

phenomenon or make

a decision

Knowledge

Integration

Explain the interconnections among

conceptual knowledge components, or

the conceptual foundation of

procedures, or the procedural

implementation of conceptual

knowledge components

Opportunities (including instructions, templates, rubrics) to self-explain, discuss, present, describe or

select their reasoning about interconnections among knowledge components, for example the

principle(s) that justify the application of a procedure.

Knowledge

Transfer

Multiple and varied contexts for

examples

Multiple and varied contexts for practice and assessment.

Opportunities for students to explain how they would use the knowledge in other contexts

*Facts are concepts with single instances

** All Use and Proxy for Use Activities develop/require procedural knowledge

Presentation and practice match objectives (knowledge components)

31

Agenda

• What evidence says about learning

• What this means for the design of instruction

• What happens when you do this for real

• [How to get outcomes aligned with real expertise]

• [“Teaching & Learning in the 21st Century” – thoughts]

32

ID can change instructional outcomes at scale

Principle Description Effect size

(s.d. units)

Multimedia Use relevant graphics and text to communicate content 1.5

Contiguity Integrate the text nearby the graphics on the screen – avoid covering or separating

integrated information

1.1

Coherence Avoid irrelevant graphics, stories, videos, media, and lengthy text 1.3

Modality Include audio narration where possible to explain graphic presentation 1.0

Redundancy Do not present words as both on-screen text and narration when graphics are

present

.7

Personalization Script audio in a conversational style using first and second person 1.3

Segmenting Break content down into small topic chunks that can be accessed t the learner’s

preferred rate

1.0

Pre-training Teach important concepts and facts prior to procedures or processes 1.3

Etc. Worked examples, self-explanation questions, varied-context examples and

comparisons, etc.

??

Source: E-learning and the Science of Instruction, Clark and Mayer, 2nd ed., 2008

33

Impact is not small!

50%

1 sd 84%!

34

Redeveloping courses at scale

Read, Write, Discuss

• Outcomes and content not

precisely aligned

• Limited demonstrations, worked

examples, and practice

• General assessment rubrics

• High reliance on discussion boards

Existing courses

Prepare, Practice, Perform

• Outcomes and content aligned

• One lesson per objective

• Demonstrations and worked examples

• Practice, feedback before assessment

• Detailed scoring guides

• Less discussion/more practice

• Standard instructor materials

• Monitoring and support for motivation

Redesigned courses

35

Content Design

Items Prepare Practice Perform

Seminar Discussion Lessons Sets

Overview

Course Level

Outcome 1

Unit

Outcome 1

Unit

Outcome 2

Unit

Outcome 3

Prepare

1

Practice

1

Perform

1

Prepare

2

Practice

2

Perform

2

Prepare

3

Practice

3

Perform

3

Lesson 1 Lesson 2 Lesson 3

Navigation

36

Narrated demonstrations

37

Explanation and demonstration of concepts

38

Practice with hints and feedback

39

Scenario-based practice

40

Prompted self-explanation

41

Motivation surveys

42

6

Overview

Seminar

Discussion

Journal: A situation in your life where the

guidelines for improving nonverbal communication

could guard against misinterpretation.

Lesson 1

Identify verbal and nonverbal elements in personal and professional

situations 1-2 hrs.

Lesson 2

Identify nonverbal communication principles in personal and professional situations 1-2 hrs.

Lesson 3

Explain instances of effective and ineffective communication in terms of

how verbal and nonverbal elements work together 2-4 hrs.

UNIT 5 Review: What is Nonverbal Communication?

2. Identify nonverbal communication principles in personal and professional situations

15

UNIT 5 Review: Lesson 2 Practice

http://www.youtube.com/watch?v=bg0kSIJZiRQ&feature=related

PART 1: Which nonverbal

communication principle is

predominant in the woman’s

reactions to her blind date?

Watch Item 2 video:

21

Overview (including Survey)

Seminar

Discussion

Explain how improving your listening skills

can increase the effectiveness of your

communication in the workplace and in your

personal life.

Lesson 1

Identify forms of nonlistening in personal and professional situations

1-2 hrs.

Lesson 2

Apply the principles of mindful listening to improve the effectiveness of communication in personal and professional situations 1-2 hrs.

UNIT 6 Preview: How Does Listening Enhance Our I.C.?

25

UNIT 6: Lesson 2 Practice

Watch the Online Dating video. Answer the three questions, referring to the scoring guide.

Online Dating

After answering a question, study the

“Compare with Expert” response.

1. From the interaction does it seem to you that Chris’s mom is actively listening during the first

third of this conversation? Why or why not?

2. Apply the principles of mindful listening to improve communication effectiveness

29

Agenda Minutes

Opening 5 P

Student Questions 10 P

Review Unit 5 10 P

View Unit 6 25 P

Preview Unit 7 5 P

Wrap Up 5 P

Poll Question

How many of you still

have questions?

Post your questions in

“Course Questions” discussion board

q Yes q No

(Link in Course Home menu)

Wrap Up

Instructor seminar materials more standardized and

aligned with online content

43

= Unit 1 = Unit 3 = Unit 6

Provide materials to monitor and support motivation

44

What happened?

Research design: quasi-experimental

Control Pilot Control Pilot Control Pilot

Interpersonal Communications 8 7 4 3 237 199

Principles of Nutrition 6 4 3 2 148 89

Medical Terminology 6 7 4 2 197 220

Total 20 18 11 7 582 508

n sections n instructors n studentsCourse

• 1,090 students (508 pilot; 582 control)

• 87% female, average age 32; average household income $20,000

• 3 courses

• 18 instructors

• 20 sections (assigned to pilot or control)

• 2 terms (Aug – Dec 2011)

45

Analysis

• Logistic regression to examine effect of course design

on student success

• Success (1 or 0): Defined as:

Pass (1 or 0)

+ Master course objectives (>=4 on 0-5 scale)

+ Stay (1 or 0)

• Controlled for variation in

• Instructor prior student success rates

• Student background variables • Age, prior education, prior GPA, tenure, household

income

• Calendar-based success variation

46

Student success: results controlling for variables

• 11% higher

success rate

• 28% increase

• Students in

redesigned

courses were

1.6 times

more likely to

be successful

Wald Chi-Square: 10.42, df=1, n=895, Sig<.001.

39%

50%

20%

30%

40%

50%

60%

70%

Control Pilot

Adju

ste

d s

tudent success r

ate

Adjusted student success rates with 95% confidence limits

47

More to do: adjusted results varied by course

• Odds of success more than doubled (2.5 times more) in Interpersonal

Communications

• 9% difference in Principles of Nutrition (not stat significant due to smaller sample)

• Small improvement in Medical Terminology due to difficulty level of early units

48

Student quote on benefits of added practice

“Something I found to be interesting was the degree of understanding between me and another individual that wasn’t in this class. A girl I had met in a previous term that has a similar degree plan but ended up in a regular medical terminology course, still we would discuss the differences and similarities between are assigned classes. During our unit 8 test she called me hysterical about all the different elements of the final tests and couldn’t seem to grasp the concept of the 1st part of the test i.e., analysis diagram, creating new terms from word roots etc. I was mystified that something that had become 2nd nature to me mainly due to the time spent every week filling out the Analysis Tables was so difficult for her to comprehend. It was at that point I realized all the griping I had done was actually the reason my level of understanding is more evolved than somebody who never experienced it.”

49

Quote from a student who previously failed

“This course was difficult for me to do. I

tried to do this course when I attended

another school and I failed it. I think the way

the course was set up and how it broke

everything down really helped me to

understand it and pass it this time. I would

not change a thing about how this course

was set up.”

50

Satisfaction (end of term survey, 5 point scale)

Student satisfaction • Lower on end of course survey in redesigned courses (mean 4.4

vs 4.8), but still greater than 4 on 5 point scale. High positivity

scores in motivation survey.

Why?

• Courses more rigorous, more work to complete; this is a

common finding in other research (e.g., Clark, 1982)

Instructor satisfaction • Higher in redesigned courses (mean 4.6 vs 4.1)

Why?

• Detailed scoring guides for assignments

• Less time in discussions – more time to monitor and

communicate with at-risk students

• Standard seminar format and content

• Student materials: structure, clarity, practice

51

Agenda

• What evidence says about learning

• What this means for the design of instruction

• What happens when you do this for real

• [How to get outcomes aligned with real expertise]

• [“Teaching & Learning in the 21st Century” – thoughts]

52

Employers actually expect job applicants to lack the

occupational/technical skills required to do the job…

• Slightly over half of all

respondents (52.8%)

expected that job applicants

would lack occupational

skills

• In healthcare, where

occupational certifications and

licensures are required, over

68% of respondents expect

that job applicants would lack

occupational skills

Do you expect job applicants to be lacking specific occupational

skills or technical skills?

March 2011 Workforce Connections, Inc. survey of employers in western Wisconsin. Over 400 employers

from all 8 counties responded to the survey. All sizes of businesses were represented with the majority of

responses coming from businesses with less than 50 employees.

53

… and end up investing significantly in training (if they

can afford it)

What area of training comprises the

bulk of your training budget?

• Of the companies that have

training budgets, 68% of the

budget is allocated towards

skills training for new workers

• “The cost of losing and replacing

an experienced paralegal is …

roughly $100,000.”1

• “Annual paralegal turnover is

nearly 50% a year in many large

firms”…. 2 (and)”… is about 28%

nationally”. 3

1 Source: American Bar Association Standing Committee on Paralegals

(2001)http://apps.americanbar.org/legalservices/paralegals/update/campbellarticle.html 2 Greene, A. and Cannon, T. (2003) Paralegals, profitability and the future of your law practice. 3 Jordan, P. D. (2001) Paralegal Studies (quoting from Bureau of Labor Statistics).

Poorly trained employees drive high

turnover rates

54

CTA lets us do better than letting experts teach

• Experts are mostly unable, unaided, to express in words to novices more

than 30% of their decision-making

- They can visualize procedures, but not cognitive decisions

• CTA gets to 70-80% of expert decision-making

- Structured interviews with objectively-determined experts

- Refined to a “gold standard” of decisions and tasks

• When coupled with well-structured training (see Kaplan Way), takes 20%+

less effort, increases student learning by 25%+ with fewer errors

• Can influence or change how professions think about themselves – but may

have to leave some “hoops” for training to gain acceptance

55

CTA methods can help learning environments work better

• CTA methods have evidence they unlock 40-50% more of experts’ skills, reduce

training time, and increase student’s confidence

• CTA provides the inputs (including task scenarios) to inform high-quality, complete,

task-centered instruction

From patent

examiner CTA:

56

• Commanding Generals

• Financial Analysts

• Trauma Surgeons

• Fire Chiefs

• Geologists

• Salespeople

• Pharmacological Researchers

• Experimental Psychologists

• Patent Examiners

• Research Librarians

• Nuclear Generator Design

Engineers

• Psychotherapists

• Chemists

• Radiological Cardiologists

• Neonatal Nurses

• Classroom Teachers

• Fighter Pilots

• SWAT Teams

• Emergency Room Teams

• Football Coaches

• Blackjack winners (21)

• Chicken Sexers

• . . .

A wide array of professions have already used CTAs

57

CTA has made real differences in training time and

learner success by investing in design up-front

Medical school surgical instruction

CTA-trained surgeons had greater gains

from pretest to post-test in less time

Also outperformed control group on patients

in every measure of performance

Kaplan University Online Faculty

CTA-based assessment instrument identified

faculty whose students achieved 5% less

Urate and .5 higher GPA on average than

others teaching the same courses.

KU is currently developing hiring tools and

training to take advantage of the strategies.

Spreadsheet training

Scores on post-test problems, and average

time to completion:

• Discovery learning: 34% - 60 minutes

• Guided demonstration: 64% - 49 minutes

• CTA: 89% - 29 minutes

Emergency and safety procedures

New course took half the time with higher

scores on the performance posttest

CTA required 85% more front-end time for

design, development, and PD

58

CTA is a systematic way to document expertise

• CTA is an interview strategy for capturing how highly successful experts perform complex tasks in a variety of settings

• Goal is to develop authentic demonstration and practice opportunities for how to perform at expert levels

• Experts are interviewed who 1) have recent (past 2-3 mo.) experience, 2) are consistently successful, and 3) are NOT trainers.

• Interviews are done with 3-4 experts to unpack their strategies; these are merged to make an efficient approach suitable for training

• A range of problem examples or performance scenarios are collected from the experts for use in instruction as well

59

Medical Assistant current course content:

X = substantial content; x = ancillary content

Pharmacology course

Diseases - human body

60

MA CTA: Identifies key tasks/skills performed by experts

Tool Skills

Clinical Skills

Patient Assessment

First Aid

Medical History

Patient exam prep

Administering Medication

Remove sutures

Vital signs

Patient Education

Prep and clean exam room

Collect samples and lab specimens

Administrative Skills

Medical records

Prior authorization

Communication

Computer use

Ethical skills

Supply maintenance

Laboratory Skills

Preparation of

medication

Preparation of

samples and lab

specimens

Course Content

Medical Law

and Bioethics

Medical Terminology

Anatomy and

Physiology

Pharmacology

Diseases of the Human Body

Medical Office

Management

Medical Coding and Insurance

Professionalism

in Health Care

Clinical Competencies

Original content New focus

• Tie to domain tasks as

identified by experts

61

MA Program: Skills addressed in new sequence

Tool Skills

Clinical Skills

Patient Assessment

First Aid

Medical History

Patient exam prep

Administering Medication

Remove sutures

Vital signs

Patient Education

Prep and clean exam room

Collect samples and lab specimens

Administrative Skills

Medical records

Prior authorization

Communication

Computer use

Ethical skills

Supply maintenance

Laboratory Skills

Preparation of

medication

Preparation of

samples and lab

specimens

New focus

• Tie to domain tasks as

identified by experts

62

Task-centered instruction

• Move from simple to increasingly difficult tasks – NOT “PBL” sink or swim

• Teach everything needed for each task

• Fade coaching/support over time

63

MA Program: Skills addressed in new sequence

• Tie to domain tasks as

identified by experts

• Repeated use of skills

across courses

B: Begin; A: Advanced; R: Reinforce

64

MA Program: New courses include previous content

Original Course Content

Proposed Course Sequence

Admin Skills 1

Clinical Skills 1

Admin Skills 2

Clinical Skills 2

Lab Skills

Medical Law and Bioethics

X x X x x

Medical Terminology

X X X X X

Anatomy and

Physiology X X X

Pharmacology X X X

Diseases of the

Human Body X X x

Medical Office Management

X X X x X

Medical Coding and Insurance

X X X

Professionalism

in Health Care X X x x X

Clinical Competencies

X X X X X

X = substantial content; x = ancillary content

• Tie to domain tasks as

identified by experts

• Repeated use of skills

across courses

• Original concepts spread

across task instruction,

not confined to courses

65

Agenda

• What evidence says about learning

• What this means for the design of instruction

• What happens when you do this for real

• [How to get outcomes aligned with real expertise]

• [“Teaching & Learning in the 21st Century” – thoughts]

66

67

Diversity?

• Absolutely

• An expectation that a wider variance of already-

mastered skills has to be accommodated

• Much more flexibility around the logistics of mastery –

schedules, life-stage, life changes, etc.

• New work to help the diverse array of learners really

understand what works for learning (Carol Dweck)

• Arguably, similar issues for faculty – learners too!

68

More innovative pedagogies and experiential learning?

• Yes – but need to be taking into account how our

learning machinery actually works (medical analogy)

-Whether you do or don’t – the real world awaits. . .

• Guided/structured experiential learning for novices

• Closer tie between each concept and applications– both

must get taught and practiced in close proximity

• Challenge: How to balance individual instructor

innovation with lessons learned from thousands?

69

More collaborative?

• Collaborating to solve a hard problem at work is not the

same as collaborating to learn

• Learning to collaborate as one does at work is a terrific

goal – but is just as hard as domain-specific objectives

• For novices, collaboration to learn too soon can be too

much cognitive load – may block mastery of key

objectives

• Group activities after mastery are terrific for cementing,

extending, generalizing

70

Supported by technology?

• Technology does NOT solve learning problems per se

(or any other problem)

• Technology takes a good (or bad) solution, and makes it

more affordable, reliable, available, data-rich, etc.

• So it is and will be a critical component of educational

systems – much more than now

• But if learning does not take precedence. . .

• A key is much more systematic use of data – to evaluate

measures of learning, processes for learning, to

personalize learning to student’s needs

• Technology (well-deployed) should help us find what

works and deploy it systematically and well

71

Changing student expectations?

• Students should have the right to expect that what they learn is deeply

tied to what experts really decide and do – variable now, most careers!

• Student’s expectations of a “good” environment are not always correct

– especially for novices in a domain

• Indeed, research shows students often think environments that work

better for them work worse, and vice-versa (Steve-Jobs-like lesson!)

• Culture can get in the way, too – beliefs about talent/learning

• Students brains are NOT “rewired now”

- It’s the same machinery (narrow working memory supported by fast

long term memory)

-There are new things driven into long-term memory, however

-The system works as it has – neurons don’t follow Moore’s law!

-E.g.: multi-tasking – can produce, but not become better

72

A tool we’re using: An evidence-based checklist –

specifications for design and quality assurance

Is the course/lesson designed for effective

knowledge acquisition and transfer?

• Learning outcomes/objectives

• Assessments

• Practice

• Presentation: Examples

• Presentation: Information

• Content chunking and sequencing

Does the course provide support for

motivation?

Does the course provide opportunities for

knowledge integration?

Are media used appropriately and efficiently?

Does instruction adapt to student's level of

knowledge and motivation?

The checklist Categories on the checklist

73

Where do items on checklist come from?

Principle Design Actions

Task-

centeredness

Include authentic

tasks that represent

the domain/learning

outcomes

Activation

Connect to learner’s

prior

experience/knowledge

/larger knowledge

structure

Demonstration

Demonstrate and give

examples of correct

performance

Application

Provide part-task and

whole-task practice

with corrective

feedback

Integration

Deepen knowledge

with opportunities for

reflection, discussion,

public performance,

exploration of real life

uses

Merrill, M. D. “First Principles of Instruction,” In C. M.

Reigeluth & A. Carr (Eds.), Instructional Design

Theories and Models III (Vol. III), 2009

First Principles of Instruction

Principle Design Actions

Multimedia Use words and graphics rather than words alone

Contiguity Place printed words near corresponding graphics; Synchronize spoken words

with corresponding graphics

Modality Present words as audio narration rather than on-screen text

Redundancy Explain visuals with words in audio OR text, not both

Coherence Avoid interesting but unnecessary material; avoid extraneous audio,

graphics, words

Personalization Use conversational rather than formal style; Use effective on-screen

coaches; Make the author visible

Segmenting Break content into bite-size segments

Pre-training Teach key concepts prior to procedures or processes

Examples

Transition from worked examples to problems via fading; Promote self-

explanation of worked-out steps; Supplement worked examples with

explanations

Practice

Mirror the job; Provide explanatory feedback; Adapt the amount and

placement of practice to job performance requirements; Transition from

examples to practice gradually

Collaboration Insufficient evidence for guidelines on social learning

Learner Control/

Navigation

Give experienced learners control; Make important instructional events the

default; Consider adaptive control; Give pacing control

Build Thinking

Skills

Use job-specific cases; Make thinking processes explicit; Define job-specific

problem-solving processes

Games and

Simulations

Match game type to learning goals; Make learning essential to progress;

Build in guidance; Promote Reflection on correct answers; Manage

complexity

Source: E-learning and the Science of

Instruction, Clark and Mayer, 2nd ed., 2008

E-Learning and Multimedia Design Principles

74

Evidence-based checklist:

Objectives, Assessment, Practice

1 Learning Outcomes/Objectives

1.1 Learning objectives are stated. 0.0

1.2 Learning objectives are stated as performance objectives, i .e., what learners will be able to DO, not what they will know. 0.0

1.3 Lesson, module, units, course, and program objectives are aligned. 0.0

1.4Learning objectives map to certification requirements or competencies or domain taxonomies/standards from professional or

accreditation bodies. 0.0

1.5 Learning objectives are based on cognitive task analysis of expert performance in the domain or profession. 0.0

SECTION SCORE 0.0

2 Assessment

2.1 Assessment tasks match learning outcomes/objectives. 0.0

2.2Assessment tasks measure mastery/acquisition of knowledge components: procedures, facts, concepts, principles, processes (one

assessment may cover multiple objectives). 0.0

2.3 Rubrics (scoring guides) guide scoring and performance of assessment tasks with open-ended response formats. 0.0

SECTION SCORE 0.0

3 Practice

3.1 Practice matches assessment. 0.0

3.2Practice tasks elicit performance to develop procedural knowledge and supportive knowledge components (facts, concepts, principles,

processes). 0.0

3.3 Rules/rubrics diagnose errors and misconceptions. 0.0

3.4 Feedback/adaptation/guidance corrects errors and misconceptions. 0.0

3.5 Practice matches transfer context, e.g., job situation. (Prompt is contextually authentic. Response is cognitively authentic). 0.0

3.6 Part-task practice precedes whole task practice. 0.0

SECTION SCORE 0.0

75

Presentation: Examples and Information

4 Presentation: Examples

4.1 Examples (demonstrations, worked examples) match practice. 0.0

4.2 Demonstrations (or worked examples) i l lustrate task performance (procedures). 0.0

4.3 Examples, stories, cases il lustrate concepts, principles, processes. 0.0

SECTION SCORE 0.0

5 Presentation: Information

5.1Descriptions and explanations cover steps to perform a task (when and how), and related knowledge components - facts, concepts,

principles, processes (what and why). 0.0

5.2 Information needed to do practice tasks is emphasized; "nice to know" information is excluded or minimized. 0.0

5.3 Information is integrated (interwoven) with examples. 0.0

SECTION SCORE 0.0

76

Chunking, Sequencing, Overviews

Do course, unit, and lesson overviews support learning and motivation?

9 Overviews

9.1Overviews include orientation (description of where each component - course, unit, lesson - fits in a larger curriculum, program, course,

process, or hierarchy of objectives). 0.0

9.2 Goals are clarified (description of the learning outcomes or objectives - what learner will be able to do by end). 0.0

9.3 Value/Reasons/Benefits/Risks are explained (inoculation against low perceived value of content and/or methods). 0.0

9.4 Connection to prior knowledge/something familiar is made or activated (story, example, analogy, questions). 0.0

9.5 Outlines describe what is to come (in the course, unit, or lesson). 0.0

SECTION SCORE 0.0

6 Content Chunking and Sequencing

6.1 Content is broken into manageable chunks/segments. 0.0

6.2 Outcomes/objectives are presented in order of application, difficulty, with prerequisites first. 0.0

6.3 For each outcome/objective, the learning sequence is Overview, Presentation (Information and Examples), Practice, Assessment. 0.0

6.4 Navigation from section to section is simple and not confusing - there is a clear sequence and clear directions. 0.0

SECTION SCORE 0.0

77

Multimedia

Are media used appropriately and efficiently?

10 Multimedia

10.1 Graphics and media are relevant and not distracting (Coherence Principle). 0.0

10.2 Text and graphics are positioned close together without scrolling (Contiguity Principle). 0.0

10.3 Visuals are explained with text or audio, not both (Redundancy Principle). 0.0

10.4 For complex graphics, audio narration is used instead of on-screen text (Modality Principle). 0.0

10.5 Students control pace of media to play, pause, forward, rewind (Pacing Control Principle). 0.0

10.6Media use is consistent with Section 508 of the Americans with Disabilities Act (e.g., non-text has text equivalent; images are tagged with

text, audio has text alternative). 0.0

10.7 Look and feel are polished; media quality is adequate. 0.0

SECTION SCORE 0.0

78

Motivation, Knowledge Integration, Personalization Does the course provide support for motivation?

7 Motivation

7.1 References are made to importance/reasons for the content and/or instructional methods used. 0.0

7.2 The tone is positive, encouraging, conversational. 0.0

7.3 Statements or stories are included to attribute success and failure to effort, not innate ability. 0.0

7.4 Statements or stories are included to prevent under-confidence/anxiety and over-confidence. 0.0

SECTION SCORE 0.0

Does the course provide opportunities for knowledge integration?

8 Integration

8.1Techniques to promote deep processing and integration of knowledge are included, for example, prompted self-reflection, self-explanation,

discussions, student presentations. 0.0

SECTION SCORE 0.0

79

References:

• Why Students Don’t Like School, Daniel Willingham – highly readable! ;-)

• Talent is Overrated, Geoffrey Colvin – highly readable! ;-)

• E-Learning and the Science of Instruction, Clark and Mayer, 2nd ed.

• “First Principles of Learning,” Merrill, D., in Reigeluth, C. M. & Carr, A. (Eds.),

Instructional Design Theories and Models III, 2009.

• How People Learn, John Bransford et al, eds.

• “Design factors for educationally effective animations and simulations,” Plass,

J.L., Homer, B.D., Hayward, E.O., J Comput High Educ (2009) 21:31–61

• “The Implications of Research on Expertise for Curriculum and Pedagogy”,

David Feldon, Education Psychology Review (2007) 19:91–110

• “Cognitive Task Analysis,” Clark, R.E., Feldon, D., van Merrienboer, J., Yates,

K., and Early, S.. in Spector, J.M., Merrill, M.D., van Merrienboer, J. J. G., &

Driscoll, M. P. (Eds.), Handbook of research on educational

communciatinos and technology (3rd ed., 2007) Lawrence Erlbaum

Associates