1

LearnLab: Bridging the Gap Between Learning Science and Educational PracticeKen KoedingerHuman-Computer Interaction & Psychology, CMUPI & CMU Director of LearnLab

2

Real World Impact of Cognitive Science

Algebra Cognitive Tutor• Based on ACT-R theory

& cognitive models of student learning

• Used in 3000 schools600,000 students

• Spin-off:

Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

3

Personalized instruction

Challenging questions

… individualization

Progress…Authentic problems Feedback within complex solutions

Cognitive Tutors: Interactive Support for Learning by Doing

4

Success ingredients• AI technology• Cognitive Task Analysis• Principles of instruction &

experimental methods• Fast development &

use-driven iteration

Cognitive Task Analysis: What is hard for Algebra students?

Story ProblemAs a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work?

Word ProblemStarting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with?

Equationx * 6 + 66 = 81.90

6

0102030405060708090100

ElementaryTeachers

MiddleSchoolTeachers

High SchoolTeachers

% Correctly ranking equations as hardest

Nathan & Koedinger (2000). An investigation of teachers’ beliefs of students’ algebra development. Cognition and Instruction.

Expert Blind Spot!

Koedinger & Nathan (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences.

Data contradicts common beliefs of researchers and teachers

7

Cognitive Tutor Algebra course yields significantly better learning

Course includes text, tutor, teacher professional development

~11 of 14 full-year controlled studies demonstrate significantly better student learning

Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

8

Success? YesDone? No!Why not?• Student achievement still not ideal• Field study results are imperfect• Many design decisions with no research

base

• Use deployed technology to collect data, make discoveries, & continually improve

9

PSLC Vision• Why? Chasm between science & ed practice

• Purpose: Identify the conditions that cause robust student learning– Educational technology as instrument– Science-practice collaboration structure

• Core Funding: 2004-2014

10

What we know about our own learning

What we do not know

You can’t design for what you don’t know!

Do you know what you know?

11

Chemistry Virtual Lab

Algebra Cognitive Tutor

Ed tech + wide use = “Basic research at scale”

=

Transforming Education R&D

• Fundamentally transform– Applied research in education– Generation of practice-

relevant learning theory

+

English Grammar Tutor

Educational Games

Ed Tech => Data => Better learning

LearnLab Thrusts

LearnLab Course Committees

13

How you can benefit from LearnLab• Research

– General principles to improve learning• Methods

– Cognitive task analysis, in vivo studies• Technology tools• People

– Masters students & projects

14

What instructional strategies work best?• More assistance vs. more challenge

– Basics vs. understanding– Education wars in reading, math,

science…

Koedinger & Aleven (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Ed Psych Review.

• Research on many dimensions– Massed vs. distributed (Pashler)– Study vs. test (Roediger)– Examples vs. problem solving

(Sweller,Renkl)– Direct instruction vs. discovery learning

(Klahr)– Re-explain vs. ask for explanation (Chi,

Renkl)– Immediate vs. delayed (Anderson vs. Bjork)– Concrete vs. abstract (Pavio vs. Kaminski)– …

15

Knowledge-Learning-Instruction (KLI) Framework: What conditions cause robust learning

LearnLab research thrusts address KLI elements

• Cognitive Factors – Charles Perfetti, David Klahr

• Metacognition & Motivation– Vincent Aleven, Tim Nokes-Malach

• Social Communication – Lauren Resnick, Carolyn Rose

• Computational Modeling & Data Mining – Geoff Gordon, Ken Koedinger

Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.

16

Results of ~200 in vivo experiments =>Optimal instruction depends on knowledge goals

17

Cognitive Task Analysis using DataShop’s learning curve tools

Without decomposition, using just a single “Geometry” KC,

Upshot: Can automate analysis & produce better student models

But with decomposition, 12 KCs for area concepts,

a smoother learning curve.

no smooth learning curve.

18

How you can benefit from LearnLab• Research

– General principles to improve learning• Methods

– Cognitive task analysis, in vivo studies• Technologies

– Tutor authoring– Language processing– Educational Data Mining

• People: Masters students & projects

19

Questions?

20

Question for you

What do you need in a learning science professional?

21

22

Extra slides

23

3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9

Cognitive Tutor Technology• Cognitive Model: A system that can solve problems in

the various ways students can

If goal is solve a(bx+c) = dThen rewrite as abx + ac = d If goal is solve a(bx+c) = d

Then rewrite as abx + c = d

If goal is solve a(bx+c) = dThen rewrite as bx+c = d/a

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

24

3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9

Cognitive Tutor Technology• Cognitive Model: A system that can solve problems in

the various ways students can

If goal is solve a(bx+c) = dThen rewrite as abx + ac = d If goal is solve a(bx+c) = d

Then rewrite as abx + c = d

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

Hint message: “Distribute a across the parentheses.” Bug message: “You need to

multiply c by a also.”

• Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing

Known? = 85% chance Known? = 45%

25

Cognitive Task Analysis Improves Instruction• Studies: Traditional instruction vs. CTA-based

– Med school catheter insertion (Velmahos et al., 2004)– Radar system troubleshooting (Schaafstal et al., 2000) – Spreadsheet use (Merrill, 2002)

• Lee (2004) meta-analysis: 1.7 effect size!

26

Learning Curves

27

Inspect curves for individual knowledge components (KCs)

Some do not =>Opportunity to improve model!

Many curves show a reasonable decline

28

DataShop’s “leaderboard” ranks alternative models100s of datasets from ed tech in math, science, & language

Best model finds 18 components of knowledge (KCs) that best predict transfer

28

Data from a variety of educational technologies & domains

29

Numberline Game

Statistics Online Course English Article Tutor

Algebra Cognitive Tutor

Model discovery across domains

3011 of 11 improvedmodels

Variety of domains& technologies

Koedinger, McLaughlin, & Stamper (2012). Automated student model improvement. In Proceedings of Educational Data Mining. [Conference best paper.]

31

Data reveals students’ achievement & motivations

We have used it to• Predict future state test scores as well

or better than the tests themselves• Assess dispositions like work ethic• Assess motivation & engagement• Assess & improve learning skills like

help seeking…

32

LearnLab courses at K12 & College Sites

• 6+ cyber-enabled courses: Chemistry, Physics, Algebra, Geometry, Chinese, English

• Data collection– Students do home/lab work

on tutors, vlab, OLI, …– Log data, questionnaires,

tests DataShop

Researchers Schools

Learn Lab

Chemistry virtual lab

Physics intelligent tutor

REAP vocabulary tutor

33

Lab experiment

In Vivo Experiment

Design Research

Randomzd Field Trial

Setting Lab School School School

Control condition Yes Yes No Yes

Focus on principle vs. on solution

(Change N things)

Scientific Principle

ScientificPrinciple

Instr. Solution

Instr. Solution

Cost/Duration $/Short $$/Medium $$/Long $$$$/Long

Bridging methodology: in vivo experiments

34

Knowledge Components• Definition: An acquired unit of cognitive

function or structure that can be inferred from performance on a set of related tasks

• Includes:– skills, concepts, schemas, metacognitive strategies,

malleable habits of mind, thinking & learning skills• May also include:

– malleable motivational beliefs & dispositions• Does not include:

– fixed cognitive architecture, transient states of cognition or affect

• Components of “intellectual plasticity”

Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.

35

General knowledge components, sense-making, motivation, social intelligencePossible domain-general KCs• Metacognitive strategy

– Novice KC: If I’m studying an example, try to remember each step

– Desired KC: If I’m studying an example, try to explain how each step follows from the previous

• Motivational belief– Novice: I am no good at math– Desired: I can get better at math by studying & practicing

• Social communicative strategy– Novice: If an authority makes a claim, it is true – Desired: If considering a claim, look for evidence for &

against it

36

What is Robust Learning?• Achieved through:

– Conceptual understanding & sense-making skills

– Refinement of initial understanding– Development of procedural fluency with

basic skills

• Measured by:– Transfer to novel tasks– Retention over the long term, and/or – Acceleration of future learning

37

KLI summary• Learning occurs in

components (KCs)• KCs vary in kind/cmplxty

– Require different kinds of learning mechanisms

• Optimal instructional choices are dependent on KC complexity

Intelligence does not improve generically

Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.

38

Conclusions• Learning & education are complex

systems

• Lots of work for learning science!

• Use ed tech for “basic research at scale”=> Bridge science-practice chasm