tuteurs m étacognitifs : supporter la métacognition par la reflexion
DESCRIPTION
Tuteurs m étacognitifs : Supporter la métacognition par la reflexion. Roger Nkambou. What is a “Cognitive Model”?. A simulation of human thinking & resulting behavior Usually used to explain or predict data on human behavior Like error rates or solution time - PowerPoint PPT PresentationTRANSCRIPT
Tuteurs métacognitifs : Supporter la métacognition par la reflexion
Roger Nkambou
What is a “Cognitive Model”?
A simulation of human thinking & resulting behaviorUsually used to explain or predict data on human behavior Like error rates or solution time
Usually implemented as a computer program that can behave like humans Often using AI knowledge representations
like semantic nets, frames, schema, production rules
What are Cognitive Models used for?
Output of basic research Explain results of psychology experiments
Guide design of software systems Have cognitive model “use” the system
Model predicts people’s time & errors(VanLehn) Redesign system to reduce time or errors
Can derive predictions without full implementation (e.g., Ethan)
As a component in an intelligent system Player in a game or training simulation Part of expert system or intelligent tutor
What is an “Intelligent Tutoring System” (ITS)?
A kind of educational softwareUses artificial intelligence techniques to Provide human tutor-like behavior Be more flexible, diagnostic & adaptive Write more general code to get more
capabilities with less effort
Components of an ITS: Interface or problem solving environment,
domain knowledge, student model, pedagogical (tutoring) knowledge
Reflective thinking & tutoring meta-cognition
Cognitive Modeling and Intelligent Tutoring Systems
Ken KoedingerVincent Aleven
Overview
ACT-R background & declarative transfer
Two studies of tutoring meta-cognition
Future: 3rd generation tutors
Different Learning Goals
From: e-Learning and the Science of Instruction : Proven Guidelines for Consumers and Designers of Multimedia Learning by Ruth Colvin Clark & Richard E. Mayer, 2002.
Corresponding Instructional Approaches
ACT-R’s declarative-procedural distinction
Declarative knowledge Includes facts, procedures that people can describe Stores inputs of perception & includes visual memory
Procedural knowledge Performance knowledge, cannot be verbalized
Procedural k “runs on hardware” Efficient
Declarative k is interpreted by procedural k Can be flexibly adapted But requires associated interpretive procedural k
Calculus Study in Declarative Transfer chapter of Singley & Anderson
What’s the difference between operator selection & operator application?What are the four training conditions in the study? What’s the same in all 4?During test (day 2) the interface is like which training condition?Is there transfer from operator … application to selection? selection to application?
Declarative Transfer Summary
Declarative k is basis for transfer b/t different uses of same knowledgeMay be short-lived & sometimes overshadowed by extended practiceNeed to search for source of analogy Can be problematic (Gick & Holyoak) Requires world knowledge & can serve well
as a learning & transfer mechanism even as young as 3 yrs old (Brown & Kane)
Overview
ACT-R background & declarative transfer
Two studies of tutoring meta-cognition
Future: 3rd generation tutors
Meta-Cognition 1: Encourage Active Declarative Processing Through Self-Explanation
Aleven, V. & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26(2)
Problem: Shallow knowledge acquisition
Variations on shallow knowledge Over-general procedural knowledge
right for wrong reason No declarative k -- cannot explain, transfer
Geometry example “Looks-equal” production rule If the goal is to find angle A
and it looks equal to angle B and angle B is D degreesThen conclude that angle A is D degrees
Example of Shallow Reasoning
Hypothesized SolutionActive processing of declarative knowledge of problem-solving principles leads to: Better detection of relevant features behind
correct inference Provides dual code for enhanced memory Less error-prone implicit procedural learning
Instructional manipulation: Ss explain steps using principles
& get feedback on explanations
Explanation Condition
Problem solving answers
Explanation by reference
Problem Solving Condition
SE Study 1 MethodBetween subjects comparison: Problem Solving vs. Explanation
Run in a Geometry class at local HSParticipants 41 high school geometry students total 24 Ss provided complete data, pre-test,
tutor, & post-test
About 7 hours of instruction Ss done when they satisfy tutor’s mastery
criteria on problem solving skills
HypothesisRequiring students to explain steps results in deeper understanding: Less shallow procedural knowledge More general declarative knowledge
Consequences: Better reason giving Near transfer as good or better Better far transfer
Pre/Post Test Items
Problem-solving items Answer - Finding unknown quantities
Items associated with deeper understanding Reason - Explain answers by citing
geometry rule Not Enough Info - Transfer items where
students are asked to judge if there is enough information to find quantities, and the answer is “No”.
Assessing transfer: “Not Enough Info” item
Assessing transfer: Incorrect over-generalization
SE Study 1 Results
Answer Items Reason Items Not Enough Info Items
0
.1
.2
.3
.4
.5
.6
.7
.8
.9
Reason
Answer Only
%
Corr
ect
Condition
Possible confounding factors in study 1: time & S prior ability
Answer-OnlyCondition
ReasonCondition
Time on Tutor 383 mins 436 mins
Prior Score 82.3 87.3
Number of Problems Solved 135 102
Neither difference is statistically significant but ... Hard to rule out alternative explanation: Explanation condition had more time & higher prior ability
Self-Explanation Study 2 Motivation
Replicate the results of Study 1, while controlling for time on task
SE Study 2 MethodBetween subjects comparison: Problem Solving vs. Explanation
Run in a Geometry class at local HSParticipants 53 students total 41 provided complete data
7 hours of instruction Time fixed, so all students spent the same
time
No time differences in Study 2
Answer-OnlyCondition
ReasonCondition
Time on Tutor 501 mins 513 mins
Number of Problems Solved 111 76
Differences between conditions cannot be attributed to differences in time on task
SE Study 2 Results
Answer Items Reason Items Not Enough Info Items.2
.3
.4
.5
.6
.7
.8
Pro
port
ion C
orr
ect
on P
ost
-Test
ReasonAnswer Only
Numerical steps Explanations Transfer items
Problem SolvingExplanation
Condition
Different instruction => different kinds of knowledge acquisition
Shallow (over-general procedural) Right answers for wrong reason,
wrong answers when pressed
Procedural Right answers with correct knowledge Efficient, fluent, but inflexible
Declarative Principles interpreted & reflectively applied Flexible, but slow & may fail in high
cognitive load situations
Extra Practice in Problem Solving => More Shallow Learning
Easy to guessitems
Hard to guessitems
.3
.4
.5
.6
.7
.8
.9
1
% C
orr
ect Explanation
Problem Solving
Condition
Shallow Procedural Knowledge vs. “Frontal” Control
Commission errors / total errors
Easier-To-Guess Harder-To-Guess.1
.2
.3
.4
.5
.6
.7
.8
.9
Explanation
Problem Solving
Condition Explanation Problem Solving
Problem Solving group jumps to incorrect conclusions
Explanation group shows more control, reflects on sufficiency of knowledge
Student Performance During Instruction
ExplanationCondition
Problem -SolvingCondition
SuccessRate
# ofSteps
SuccessRate
# ofSteps
All problemsolvingsteps
51% 237 56% 457
First 237steps
51% 237 51% 237
Rest ofsteps
- - 62% 220
Explanationsteps
55% 236 - -Problem solving group appears better at end of tutoring. But, not better on post-test!
Shallow procedural knowledge acquisition => lack of transfer
Estimating Acquisition of Different Knowledge Types
Knowledge Type ExplanationCondition
ProblemSolving
Condition
Shallow ProceduralKnowledge
0.58 0.68
Correct ProceduralKnowledge
0.30 0.42
DeclarativeKnowledge
0.32 0.12
Predicts Performance on Different Test Items ...
Explanation Condition Problem-SolvingCondition
Type of Test Item Variable Actual Predicted Actual Predicted
Numeric, Easier-to-Guess E 0.76 0.80 0.84 0.84
Numeric, Harder-to-Guess H 0.54 0.52 0.49 0.49
Not Enough Information N 0.58 0.61 0.41 0.41
Explanation R 0.51 0.49 0.34 0.34
ImplicationsWhen Ss explain they learn more & learn with greater understanding: better explanations of answers better on harder-to-guess test items better on transfer questions
Possible to achieve benefits of self-explanation with simple manipulationFuture work: system with which students can explain in their own words
Meta-Cognition 2: Supporting Error Detection & Self-Correction
PhD student Santosh Mathan
Benefits of Immediate Feedback
Supports efficient skill acquisition Eliminates
floundering
LISP Tutor study Faster learning Same post-test
65432100
250
500
750
1000
1250
1500
Immediate Feedback
Error Flagging
Demand FeedbackNo Feedback
Tutor Lesson
Criticisms of Immediate Feedback
Qualitative Basis Human tutors may wait (Merrill, 1995) But, just because humans do it ...
Empirical basis Benefits of delayed feedback in motor learning
Schmidt et al., 1988
Some cognitive studies Transfer (Lee, 1992) Retention (Schooler & Anderson, 1985)
Recasting Delayed vs. Immediate Feedback Debate
Debate cast in terms of latencyAlternative: What is the “model of desired performance”?Expert Model
immediate error correction emphasizes generative skills
Intelligent Novice Model allows errors, guides students through error
detection & correction emphasizes generative & evaluative skills
Domain of study
Cell referencing in Excel spreadsheet programming
“Glass ceiling” in natural spreadsheet use & skill acquisition
Expert Feedback
Expert Feedback
Intelligent Novice Feedback
Intelligent Novice Feedback
Intelligent Novice Feedback
Participants
48 participants recruited from a temporary employment agency
All had general computer experience
No Excel experience
Instruction, transfer & retention testing
Day 1
Day 2
.
.
.
.
Day 3
Pre Test Declarative Procedural Post Test
Procedural Post Test
Transfer Pre Test Procedural Post Test
90 min
50 min
30 min
8 days later
Kinds of Pre & Post Tests
Prior experience tests Computer experience questionnaire Algebra word problems
Excel coding testExcel concept testTransfer coding task More complex with novel demands
Results
Students using intelligent novice model tutor significantly outperformed students using expert-model tutor on all measures Coding Concepts Retention Transfer
Coding Performance
F = 4.23, p < .05
75.90%
85.20%
0%
20%
40%
60%
80%
100%
Expert Intelligent Novice
Conceptual Performance
F =4.06, p < .05
66.60%72.90%
0%
20%
40%
60%
80%
100%
Expert Intelligent Novice
Retention Session Performance
F = 4.07, p < .05
72.50%
81.20%
0%
20%
40%
60%
80%
100%
Expert Intelligent Novice
Transfer Performance
F = 5.662, p < .03
59.79%
74.31%
0%
20%
40%
60%
80%
100%
Expert Intelligent Novice
Learning Curves: Difference Between Conditions Emerges Early
6 production rule model - surface feature model that separates surface differences
Opportunities to apply a production rule
Num
ber
of a
ttem
pts
at a
ste
p
Learning Curves: Deeper Generalizations
4 production rule model - deep feature model that merges surface feature differences
Opportunities to apply a production rule
Num
ber
of a
ttem
pts
at a
ste
p
ImplicationsIntelligent novice (IN) model feedback produces: better learning outcomes, retention, &
transfer
On-line data shows effects when coded by production rulesDifference in declarative encoding EX: shallow declarative encoding IN: more general declarative encoding
Future Contrast IN & delayed feedback
Summary: Cognitive Tutors & Meta-Cognition
Cognitive Tutors are experimental instrumentation Control instruction, long duration,
fine grain pre-coded learning data
Can support meta-cognitive processes to enhance transfer & retention Simple self-explanation by reference works Can model intelligent novice error detection
& self-correction skills Reinterpretation of feedback timing debate
Summary: Cognitive Tutors & Meta-Cognition
Can support development of both fluent domain experts & flexible "intelligent novices" Not just better, but different learning by supporting meta-cognitive processesDifferent knowledge is acquired: Less shallow procedural knowledge More declarative knowledge & interpretive
procedures
Overview
ACT-R background & declarative transfer
Two studies of tutoring meta-cognition
Future: 3rd generation tutors
3 Generations of Tutors
1st Generation -- Been on the market Underlying technology: Hypertext & Behaviorism Pedagogy: Didactic feedback on answers.
2nd Generation -- Emerging in the market Technology: Artificial Intelligence & Cognitive Psychology Pedagogy: Assistance on problem solving steps, not just
final answers
3rd Generation -- Emerging in the lab Technology: Natural language processing, reactive
planning, Cost-effective pedagogical-content assessment Pedagogy: Knowledge constructing dialogs
The first generation:Computer Aided Instruction (CAI)Underlying technology: Hypertext & BehaviorismPedagogy: Didactic feedback on answers.Example:
Solve 2+2x=12 Multiplication has ahigher precedencethan addition, so 2+2xis the same as 2+(2x),not (2+2)x. Try again.x=7
x=3
x=5
OK
OK
Excellent!
First Generation Tutor Example
Second generation:Intelligent Tutoring Systems (ITS)Underlying technology: Artificial intelligence & cognitive psyPedagogy: Assistance on problem solving steps, not just final answersExample:
Tutor: Solve 2+2x=12 Student: <enters 4x=12> Tutor: Not quite. Try again. Student: <clicks on “hint” button> Tutor: Think about operator
precedence. Student: <enters 2x=12-2> Tutor: Good!
2 + 2x = 12 4x = 122x = 12 - 2
Student’s workspace:
Tutor:
Good!
Hint
Algebra Cognitive Tutor (CL, Inc)
Technology: Reactive planning & Natural language processingPedagogy: Knowledge constructing dialogsExample:
Tutor: Solve 2+2x=12 Student: 4x=12 Tutor: Should this equation have the
same solution as the first one? Student: Yes. Tutor: The solution to 4x=12 is 3,
so let’s check for an error by trying x=3 in 2+2x=12.
Student: 2+2*3=2+6=8 oops! Tutor: Right! Now look at the
arithmetic steps you did …
Third generation
2+2x=124x=12
Student’s workspace:
Dialog:
S: 2+2*3=2+6=8 oops!T: Right! Now look...
Hint
The nested loops of CAI (1st generation tutors)
For each chapter in curriculumRead chapterFor each exercise Attempt answer Get feedback & hints on answer; try again If mastery is reached, exit loop
Take a test on chapter
The nested loops of ITS (2nd gen)
For each chapter in curriculumRead chapterFor each exercise For each step in solution
Student attempts step Get feedback & hints on step; try again
If mastery is reached, exit loop
Take a test on chapter
The nested loops of dialogue-based tutors (3rd gen)
For each chapter in curriculumRead chapterFor each exercise For each step in solution
Student attempts step If incorrect, for each inference in a directed
line of reasoning Elicit the inference from student Hint, prompt, pump; try again If S completes step, exit loop
If mastery has been reached, exit loop
Take a test on chapter
Limitations of 2nd-generation tutors
Better than classroom instruction, but not as good as human tutors! Human tutors 2 better (standard deviations or
“sigma”) than classroom instruction The best 2nd-generation tutors are 1 better
Do not always lead to deep understanding Symptoms of shallow learning
Lack of transfer to novel problems Inability to explain / carry on coherent abstract
conversation about the domain A problem for many instructional methods!
Third-Generation Tutors
Knowledge construction dialogs“there is something about conversational dialog
that plays an important role in learning”.
Better theory about how to get students to learn:“Good tutors tell less and ask more.”They guide students as they construct new
knowledge. Help them make abstractions, connections.
Third-generation tutors Research agenda
Empirical When and why is tutorial dialog effective?
Technical Develop systems that support students in
constructing knowledge through dialog Evaluate the effectiveness of the systems
Goal is to rival or surpass the effectiveness of human tutors
Examples of 3rd Generation Tutors
Andes/Atlas: Dialog instead of hintsWhy/Atlas: Dialog critiques of an essay-long physics explanationCIRCSIM: Dialog to correct errors in students’ predictions about physiological causalityAutoTutor: Dialog about computer literacyGeometry explanation: Expressed in dialog instead of menu choice & feedbackMs. Lindquist: Dialog about methods for analyzing algebra word problemsCATO: Courtroom dialog to teach heuristics for citing cases
Andes/Atlas: Dialogue replaces hint sequences
Andes: If you are moving in a straight line and accelerate in the same direction, does your velocity increase or decrease?
You: increase
Andes: You’ve drawn the acceleration of the elevator in the same direction as the velocity. Is the velocity of the elevator increasing?
Why/Atlas
CIRCSIMMartha Evens, Reva Freedman, Michael Glass, Yujian Zou, et al., Illinois Institute of Technology
Domain: physiology (negative feedback loop that controls blood pressure)Emphasis: dialog to correct errors in students’ predictions about physiological causalityShort answer questions, hint strategiesModeled after expert human tutors
CIRCSIM-Tutor User Interface
Problem: Pacemaker malfunctions, increasing to 120 beats/min.
T> What variable is affected by HR?S> Cardiac Output.T> But you predicted that HR increases and CO decreases.S>
DR RR SSCentral Venous Pressure -Inotropic State 0Stroke Volume -Heart Rate +Cardiac Output -Total Peripheral Resistance 0Mean Arterial Pressure +
MeanArterial
Pressure
TotalPeripheralResistance
ArteriolarResistance
ArteriolarMuscleTone
ArteriolarDiameter
StrokeVolume
CentralVenous
Pressure
InotropicState
Alpha-Blocker(drug)
Pacemaker Rate Up(or Down)
IntracellularCa++
Concentration
Beta-Blocker(drug)
Transfusion(or Hemorrhage)
Blood VolumeCentral Blood
Volume
CardiacOutput
HeartRate
Sino-AtrialNodeRate
NervousSystem
Response
BaroreceptorPressure
A Family of Possible Dialogs with CIRCSIM
I haveno idea
TPR isneurally
controlled
Sympatheticvasoconstriction
Nervoussystem
Radius ofarterioles
Right
Right.TPR isneural.
And whatcontrols
that?
Nervoussystem
Right
(3) So what must be the value of TPR?
(1) Can you tell me what controls TPR?
(2) Remember that we’re in DR now.
<wrong>
POP
AutoTutorArt Graesser, Derek Harter, Peter Wiemer-Hastings, Natalie Person, et al., University of Memphis
Domain: Computer LiteracyEmphasis: Simulating the average human tutor Mixed-initiative dialog with deep-reasoning questions Interface: Talking face, with gestures, intonation
Dialog Moves Pumping Prompt, hint, assert
Feedback Back channeling Pedagogical feedback (evaluation of answer) Corrective feedback
Responding to student initiative
AutoTutor with Marco Talking Head
An Example Tutorial Dialog with AutoTutorTUTOR-1: Alright, let's go on. You want to take photos and send them over the Internet. Consider this problem. What hardware do you need to take photos and send them over the Internet?
STUDENT-1: First of all you need to have access to the internet
TUTOR-2: What else?STUDENT-2: a cameraTUTOR-3: Yeah. For what type of connection do you need a network card?
STUDENT-3: network cardTUTOR-4: Okay. Now, if you have access to the Internet through a network card, then your connection is …
STUDENT-4: I don't know.TUTOR-5: Is direct. Okay. A network card is needed if you have a direct connection to the Internet. Okay. How does the user get hooked up to the internet?
STUDENT-5: by a plug.
TUTOR-6: Once again, a network card is needed if you have a direct connection to the Internet. Okay, what does a modem do?
STUDENT-6: connects network card and internet
TUTOR-7: Okay. Try this. A modem is required when there is …
STUDENT-7: a connectionTUTOR-8: a dial up connection. ...TUTOR-30: Let’s review. To send your photos on the Internet, you need either a digital camera or a regular camera to take the photos. If you use a regular camera, you need a scanner to scan them onto a computer disk. If you have a direct connection to the Internet, then you need a network card. A modem is needed if you have a dial up connection.
Why might natural language dialog enhance learning (or not)?Because it’s natural…
Draws student more into thinking into higher levels Leading student on to get more help whereas
student has to press more in 2nd gen
More tutor interventionWith speech processing Easier to speak than to write,
easier to listen than to read
S can express ideas informally in NL Reveals partial knowledge
Goal: Help students learn with greater understanding by tutoring at the explanation level
From: Guided learning by doingTo: Guided learning by doing and explaining
Motivated by Cognitive Science literature on the “Self-Explanation Effect”How can a computer tutor best support Self-Explanation? Have students explain in their own words? Is feedback on explanations necessary?
A 3rd-Generation Tutoring System: The Geometry Explanation Tutor
Study 1: Explanation ‘by reference’
Tutor feedback indicates whether explanation is correct.
Study 2:How about explaining in one’s own words, but without tutor feedback on explanations?
Students produce very few good explanations ...
Hypothesis: Students learn best when they explain in their own words and tutor help them to refine explanations, through dialog.
Geometry Explanation Tutor—A Tutorial Dialog System to Support Self-Explanation
Example Dialogue with the Geometry Explanation Tutor
Student: angles are equalTutor: Are any two angles congruent? What tells
you that there are two congruent angles here?Student: angles in a triangle are equalTutor: That is a bit imprecise. Are ALL angles in
ALL triangles congruent?Student: angles in an isosceles triangle are equalTutor: Correct, except for one thing: are all three
angles in an isosceles triangle congruent?Student: base angles in an isosceles triangle are
equalTutor: CORRECT
USER INTERFACE
LCFLEX PARSER
FEATURE STRUCTURE
UNIFIER
LOGIC SYSTEM (Loom)
PRODUCTION ENGINE
COGNITIVE MODEL
SEMANTIC REPRESENTATION
of Explanation
FEATURE STRUCTURES
KNOWLEDGE BASE —Ontology
& Explanation Hierarchy
GRAMMAR & LEXICON
COGNITIVE TUTOR
NLU COMPONENT
STATISTICAL CLASSIFIER
Student Explanation
Feedback or Help Message
Detailed Classification of
Explanation
Ballpark Classification of
Explanation
(Numerical)Answer or
Hint Request
STUDENT MODEL
Pedagogical Content Knowledge: Explanation Hierarchy
Hierarchy of Partial ExplanationsExcerpt — Isosceles Triangle TheoremUNKNOWN
CONGR-ANGLES“The angles are congruent.”BASE-ANGLES
“These are base angles.”
BASE-ANGLES-CONG“Base angles are
congruent.”
CONGR-ANGLES-IN-TRI“Angles in a triangle are
congruent.”
TRI-BASE-ANGLES“Base angles in a triangle
are congruent.”
CONGR-ANGLES-IN-ISOS-TRI
“Angles of an isosceles triangle are congruent.”
ISOS-TRI-BASE-ANGLES“Base angles in an isosceles
triangle are congruent.”
ANGLES-OPP-SIDES“Angles opposite the sides are
congruent.”
ANGLES-OPP-CONGR-SIDES“Angles opposite congruent
sides are congruent.”
ISOS-TRIANGLE“The angles opposite congruent sides in
an isosceles triangle are congruent.”
OPPOSITE-ANGLES“Opposite angles are
congruent.”
Why Might Natural Language Self-Explanations Assist Learning?“There is something about NL dialog that is right ...”It is good for students to explain in their own words …
But why?Natural language explanation requires recall, not recognition.Articulating forces attention to relevant features.Verbal learning and visual learning create “dual codes” in memory.Natural language allows for flexible expression of partial knowledge
Students can show what they do know Tutor can help student construct what they do not know
Help can come in smaller portions Tutor can support alternative developmental pathways to knowledge
construction
3rd Generation Tutor Summary
3 Generations of tutors differ in their underlying technology, psychological theory, and methods for developmentShallow learning can occur when students do not encode relevant features of the taskCIRSIM, AutoTutor, Ms. Lindquist, and the Geometry Explanation Tutor are examples of 3rd generation tutorsIntuitively, natural language dialog seems powerful for learning, but research is exploring when/why