learnlab : bridging the gap between learning science and educational practice
DESCRIPTION
LearnLab : Bridging the Gap Between Learning Science and Educational Practice. Ken Koedinger Human-Computer Interaction & Psychology, CMU PI & CMU Director of LearnLab. Real World Impact of Cognitive Science. Algebra Cognitive Tutor - PowerPoint PPT PresentationTRANSCRIPT
1
LearnLab: Bridging the Gap Between Learning Science and Educational PracticeKen KoedingerHuman-Computer Interaction & Psychology, CMUPI & CMU Director of LearnLab
2
Real World Impact of Cognitive Science
Algebra Cognitive Tutor• Based on ACT-R theory
& cognitive models of student learning
• Used in 3000 schools600,000 students
• Spin-off:
Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.
3
Personalized instruction
Challenging questions
… individualization
Progress…Authentic problems Feedback within complex solutions
Cognitive Tutors: Interactive Support for Learning by Doing
4
Success ingredients• AI technology• Cognitive Task Analysis• Principles of instruction &
experimental methods• Fast development &
use-driven iteration
Cognitive Task Analysis: What is hard for Algebra students?
Story ProblemAs a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work?
Word ProblemStarting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with?
Equationx * 6 + 66 = 81.90
6
0102030405060708090100
ElementaryTeachers
MiddleSchoolTeachers
High SchoolTeachers
% Correctly ranking equations as hardest
Nathan & Koedinger (2000). An investigation of teachers’ beliefs of students’ algebra development. Cognition and Instruction.
Expert Blind Spot!
Koedinger & Nathan (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences.
Data contradicts common beliefs of researchers and teachers
7
Cognitive Tutor Algebra course yields significantly better learning
Course includes text, tutor, teacher professional development
~11 of 14 full-year controlled studies demonstrate significantly better student learning
Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.
8
Success? YesDone? No!Why not?• Student achievement still not ideal• Field study results are imperfect• Many design decisions with no research
base
• Use deployed technology to collect data, make discoveries, & continually improve
9
PSLC Vision• Why? Chasm between science & ed practice
• Purpose: Identify the conditions that cause robust student learning– Educational technology as instrument– Science-practice collaboration structure
• Core Funding: 2004-2014
10
What we know about our own learning
What we do not know
You can’t design for what you don’t know!
Do you know what you know?
11
Chemistry Virtual Lab
Algebra Cognitive Tutor
Ed tech + wide use = “Basic research at scale”
=
Transforming Education R&D
• Fundamentally transform– Applied research in education– Generation of practice-
relevant learning theory
+
English Grammar Tutor
Educational Games
Ed Tech => Data => Better learning
LearnLab Thrusts
LearnLab Course Committees
13
How you can benefit from LearnLab• Research
– General principles to improve learning• Methods
– Cognitive task analysis, in vivo studies• Technology tools• People
– Masters students & projects
14
What instructional strategies work best?• More assistance vs. more challenge
– Basics vs. understanding– Education wars in reading, math,
science…
Koedinger & Aleven (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Ed Psych Review.
• Research on many dimensions– Massed vs. distributed (Pashler)– Study vs. test (Roediger)– Examples vs. problem solving
(Sweller,Renkl)– Direct instruction vs. discovery learning
(Klahr)– Re-explain vs. ask for explanation (Chi,
Renkl)– Immediate vs. delayed (Anderson vs. Bjork)– Concrete vs. abstract (Pavio vs. Kaminski)– …
15
Knowledge-Learning-Instruction (KLI) Framework: What conditions cause robust learning
LearnLab research thrusts address KLI elements
• Cognitive Factors – Charles Perfetti, David Klahr
• Metacognition & Motivation– Vincent Aleven, Tim Nokes-Malach
• Social Communication – Lauren Resnick, Carolyn Rose
• Computational Modeling & Data Mining – Geoff Gordon, Ken Koedinger
Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.
16
Results of ~200 in vivo experiments =>Optimal instruction depends on knowledge goals
17
Cognitive Task Analysis using DataShop’s learning curve tools
Without decomposition, using just a single “Geometry” KC,
Upshot: Can automate analysis & produce better student models
But with decomposition, 12 KCs for area concepts,
a smoother learning curve.
no smooth learning curve.
18
How you can benefit from LearnLab• Research
– General principles to improve learning• Methods
– Cognitive task analysis, in vivo studies• Technologies
– Tutor authoring– Language processing– Educational Data Mining
• People: Masters students & projects
19
Questions?
20
Question for you
What do you need in a learning science professional?
21
22
Extra slides
23
3(2x - 5) = 9
6x - 15 = 9 2x - 5 = 3 6x - 5 = 9
Cognitive Tutor Technology• Cognitive Model: A system that can solve problems in
the various ways students can
If goal is solve a(bx+c) = dThen rewrite as abx + ac = d If goal is solve a(bx+c) = d
Then rewrite as abx + c = d
If goal is solve a(bx+c) = dThen rewrite as bx+c = d/a
• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction
24
3(2x - 5) = 9
6x - 15 = 9 2x - 5 = 3 6x - 5 = 9
Cognitive Tutor Technology• Cognitive Model: A system that can solve problems in
the various ways students can
If goal is solve a(bx+c) = dThen rewrite as abx + ac = d If goal is solve a(bx+c) = d
Then rewrite as abx + c = d
• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction
Hint message: “Distribute a across the parentheses.” Bug message: “You need to
multiply c by a also.”
• Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing
Known? = 85% chance Known? = 45%
25
Cognitive Task Analysis Improves Instruction• Studies: Traditional instruction vs. CTA-based
– Med school catheter insertion (Velmahos et al., 2004)– Radar system troubleshooting (Schaafstal et al., 2000) – Spreadsheet use (Merrill, 2002)
• Lee (2004) meta-analysis: 1.7 effect size!
26
Learning Curves
27
Inspect curves for individual knowledge components (KCs)
Some do not =>Opportunity to improve model!
Many curves show a reasonable decline
28
DataShop’s “leaderboard” ranks alternative models100s of datasets from ed tech in math, science, & language
Best model finds 18 components of knowledge (KCs) that best predict transfer
28
Data from a variety of educational technologies & domains
29
Numberline Game
Statistics Online Course English Article Tutor
Algebra Cognitive Tutor
Model discovery across domains
3011 of 11 improvedmodels
Variety of domains& technologies
Koedinger, McLaughlin, & Stamper (2012). Automated student model improvement. In Proceedings of Educational Data Mining. [Conference best paper.]
31
Data reveals students’ achievement & motivations
We have used it to• Predict future state test scores as well
or better than the tests themselves• Assess dispositions like work ethic• Assess motivation & engagement• Assess & improve learning skills like
help seeking…
32
LearnLab courses at K12 & College Sites
• 6+ cyber-enabled courses: Chemistry, Physics, Algebra, Geometry, Chinese, English
• Data collection– Students do home/lab work
on tutors, vlab, OLI, …– Log data, questionnaires,
tests DataShop
Researchers Schools
Learn Lab
Chemistry virtual lab
Physics intelligent tutor
REAP vocabulary tutor
33
Lab experiment
In Vivo Experiment
Design Research
Randomzd Field Trial
Setting Lab School School School
Control condition Yes Yes No Yes
Focus on principle vs. on solution
(Change N things)
Scientific Principle
ScientificPrinciple
Instr. Solution
Instr. Solution
Cost/Duration $/Short $$/Medium $$/Long $$$$/Long
Bridging methodology: in vivo experiments
34
Knowledge Components• Definition: An acquired unit of cognitive
function or structure that can be inferred from performance on a set of related tasks
• Includes:– skills, concepts, schemas, metacognitive strategies,
malleable habits of mind, thinking & learning skills• May also include:
– malleable motivational beliefs & dispositions• Does not include:
– fixed cognitive architecture, transient states of cognition or affect
• Components of “intellectual plasticity”
Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.
35
General knowledge components, sense-making, motivation, social intelligencePossible domain-general KCs• Metacognitive strategy
– Novice KC: If I’m studying an example, try to remember each step
– Desired KC: If I’m studying an example, try to explain how each step follows from the previous
• Motivational belief– Novice: I am no good at math– Desired: I can get better at math by studying & practicing
• Social communicative strategy– Novice: If an authority makes a claim, it is true – Desired: If considering a claim, look for evidence for &
against it
36
What is Robust Learning?• Achieved through:
– Conceptual understanding & sense-making skills
– Refinement of initial understanding– Development of procedural fluency with
basic skills
• Measured by:– Transfer to novel tasks– Retention over the long term, and/or – Acceleration of future learning
37
KLI summary• Learning occurs in
components (KCs)• KCs vary in kind/cmplxty
– Require different kinds of learning mechanisms
• Optimal instructional choices are dependent on KC complexity
Intelligence does not improve generically
Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science.
38
Conclusions• Learning & education are complex
systems
• Lots of work for learning science!
• Use ed tech for “basic research at scale”=> Bridge science-practice chasm