assessment embedded in step- based tutors (sbts) cpi 494 feb 12, 2009 kurt vanlehn asu
TRANSCRIPT
![Page 1: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/1.jpg)
Assessment embedded in step-based tutors (SBTs)
CPI 494 Feb 12, 2009
Kurt VanLehn
ASU
![Page 2: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/2.jpg)
Outline
Benefits of assessment embedded in SBT Synthesis of several common ITS assessment
methods– Terminology– Where does assessment fit into an ITS?– A synthesis
![Page 3: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/3.jpg)
Why should step-based tutors (SBT) be better than multiple choice tests (MCT) for assessment?
Why is a 30 minute task (SBT) better than a 1 minute task (MCT)?– Authenticity, validity, prediction of valued tasks– “teaching to the test” becomes valuable
Why is seeing the steps better than seeing just the answer?– Assess problem solving strategy– Assess meta-cognitive strategy (e.g., use of hints)
![Page 4: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/4.jpg)
Why is embedding assessment in daily homework or seatwork better than monthly in-class testing?
Frees up time for teaching Can’t be absent on day of test “doesn’t test well” and “only good at tests” are
no longer excuses No test anxiety No cramming Instructors can see changes in competence daily
![Page 5: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/5.jpg)
Homework is unsupervised but testing is supervised. How does that affect assessment?
What if students refer to textbook, web, friends or instructor?– Sequestered problem solving is inauthentic– Assessment may improve if tutor can “see” such
references What about cheating?
– Technology can thwart most copying– Would have to hire a friend to do almost all of one’s
homework
![Page 6: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/6.jpg)
Outline
Benefits of embedding assessments in SBTs Synthesis of several common ITS assessment
methods– Terminology– Where does assessment fit into an ITS?– A synthesis
Next
![Page 7: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/7.jpg)
Definition: Knowledge component
A piece of knowledge Generic – can be applied in to multiple tasks Any form: rule, principle, concept, fact,
procedure… Examples:
– Two angles are called “supplementary” if….– To buckle your seat belt, insert the …– The gravitational constant of earth, g, is 9.8 m/s2.
![Page 8: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/8.jpg)
Definition: Mastery of a knowledge component (KC)
No more training needed– E.g., The ITS need no longer pick problems that
involve it Typical signs of mastery
– Applies the KC when should– Applies the KC only when should– Applies without referring to text, notes, …– Can explain the KC, when to apply, etc.
![Page 9: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/9.jpg)
Main goal of assessment
Never have enough data – E.g., typically do not have student’s explanations
Must estimate P(mastery) for every knowledge componentgiven only steps as data
![Page 10: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/10.jpg)
Definition: The response pattern of a student’s step
How a step was accomplished by the student E.g., 4 response patterns:
(S = student; T = tutoring system)
1. S: Correct.
2. S: Error; T: Hint; S: Correct.
3. S: Help; T: Hint; S: Correct.
4. S: Help; T: Hint; S: Help; T: Hint; S: Help; T: Bottom out hint; S: Correct
![Page 11: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/11.jpg)
Definition: The relevant knowledge components of a student’s step
The knowledge components involved in deriving the correct step– given the information available at the time the step
was done The knowledge the student should apply to
generate this step The topic of the hints (if any were given)
![Page 12: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/12.jpg)
Outline
An introduction to a common type of ITS– Terminology– Three examples
Synthesis of several common ITS assessment methods– Terminology (KC, mastery, response pattern, KCs
relevant to a step)– Where does assessment fit into an ITS?– A synthesis
Next
![Page 13: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/13.jpg)
The 3 major computations of intelligent tutoring systems
P(mastery) for each knowledge component
Problem to be solved
All correct steps in all orders
Response pattern for each student step
Expert
Assessor
Helper
![Page 14: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/14.jpg)
The expert’s computation Expert can be humans
or an expert system Solve the problem
in all acceptable ways Record steps taken Record knowledge
components used at each step
P(mastery) for each knowledge component
Problem to be solved
All correct steps in all orders
Response patterns for each student step
Expert
Assessor
Helper
![Page 15: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/15.jpg)
The helper’s computation
When the student enters a step, match it to a correct step
Give feedback & hints as necessary
Record response pattern
P(mastery) for each knowledge component
Problem to be solved
All correct steps in all orders
Response patterns for each student step
Expert
Assessor
Helper
![Page 16: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/16.jpg)
The assessor’s computation Given
– Reponse patterns for each step taken by the student
– Old P(mastery) for each knowledge component
Calculate– New P(mastery)
P(mastery) for each knowledge component
Problem to be solved
All correct steps in all orders
Response patterns for each student step
Expert
Assessor
Helper
![Page 17: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/17.jpg)
Outline
An introduction to a common type of ITS– Terminology– Three examples
Synthesis of several common ITS assessment methods– Terminology– Where does assessment fit into an ITS?– A synthesis Next
![Page 18: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/18.jpg)
Some assessment methodsKnowledge tracing (Corbett & Anderson)-- Only two response patterns: “S:Correct” & “other”-- Only one relevant knowledge component per step
SMART (Shute)-- Multiple response patterns
Andes (VanLehn et al.)-- Multiple knowledge components per step
Today: A synthesis of the above
DT tutor (Murray)-- Dynamic Bayesian networks
![Page 19: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/19.jpg)
The problem: Given step responses & prior P(mastery), calculate posterior P(mastery)
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 1
S: error; T: Hint;
S: Correct
Step 2
S: Correct
Step 3
S: Help;T: Hint;S: Help;T: Hint;S: Help;
T: B.O. Hint;S: Correct
.05
.05
.11
.10
.05
.75
.12
.50
.99
.05
Prior P(mastery) for each knowledge component (KC)
Posterior P(mastery) for each knowledge component
![Page 20: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/20.jpg)
A dynamic Bayesian network represents that P(mastery) changes during steps
Bayesiannetwork
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Bayesian network
Step 1 Step 2
P(mastery) of knowledge component 4 at time 1
P(mastery) of knowledge component 4 at time 2
![Page 21: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/21.jpg)
Assume P(mastery) changes only for relevant knowledge components
Bayesian network
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4Bayesian network
Step 1Involves only KC2 and KC3
Step 2Involves only KC1 and KC4
![Page 22: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/22.jpg)
Knowledge tracing: Assumes just one relevant knowledge component per step
Bayesian network
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4 Bayesian network
Step 1Involves only
KC2
Step 2Involves only
KC4
![Page 23: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/23.jpg)
Knowledge tracing: Network topology and conditional probability tables
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 1 Step 2
Step response
Step responseKC2 mastered: Yes No
S: Correct .92 .10
Other .08 .90
Was mastered: Yes No
Mastered 1 .47
Not mastered 0 .53
P(Step response not correct | KC2 mastered)
Called “slip parameter”
P(Step response correct | KC2 not mastered)
Called “guess parameter”
P(Mastered | Was not mastered, relevant to step) Called “acquisition parameter”
![Page 24: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/24.jpg)
Need to allow multiple relevant knowledge components per step
KC1
KC3
KCn
KC2
KC4
Step response
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 1 Step 2
Step response
Use AND node to represent that the step response depends only on
whether or not all relevant knowledge components are mastered
![Page 25: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/25.jpg)
The student’s meta-cognitive strategy also affects the student’s step response
KC1
KC3
KCn
KC2
KC4
Step response
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 1 Step 2
Step response
Meta-cognitive strategy
![Page 26: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/26.jpg)
Conditional probability table for the first “Step response” node
Meta-cognitive strategy Learn Perform
Both KC2 and KC3 mastered? Yes No Yes No
S: Correct .99 .50 .99 .10
S: Error; T: Hint; S: Correct .01 .35 .00 .01
S: Help;… T: Bottom out hint; S: Correct .00 .01 .01 .85
All other response patterns .00 .14 .00 .04
P(step response was “S:Correct” | Meta-cognitive strategy was “Learn”,
Not all knowledge components mastered before step
Strong tendency to abuse help when the meta-cognitive strategy is “Perform” and some relevant knowledge component is not mastered yet.
![Page 27: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/27.jpg)
Need to assume new P(mastery) depends on the step response as well as the old P(mastery)
KC1
KC3
KCn
KC2
KC4
Step response
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 1 Step 2
Step response
Meta-cognitive strategy
![Page 28: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/28.jpg)
Conditional probability table for a “knowledge component” node
Step response: S:Correct 1 hint Bottom out hint Other
Mastered before? Yes No Yes No Yes No Yes No
Mastered after step 1.0 .50 .95 .30 .90 .01 .55 .05
Not mastered after step 0.0 .50 .05 .70 .10 .99 .45 .95
![Page 29: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/29.jpg)
How the network should be used
Given log file – all steps taken by the student Construct the whole network – all steps in all problems Enter prior P(mastery) on knowledge components at
time 1 Clamp all step responses Update – a huge computation Read out P(mastery) on knowledge components after
last step.
![Page 30: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/30.jpg)
How the network is actually used Step 1
– Construct network– Enter prior P(mastery)– Clamp step 1 response– Update– Read out P(mastery)
Step 2– Construct network– Enter prior P(mastery)– Clamp step 2 response– Update– Etc.
KC1
KC3
KCn
KC2
KC4
Step response
KC1
KC3
KCn
KC2
KC4
Step 1
Meta-cognitive strategy
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 2
Step response
Meta-cognitive strategy
![Page 31: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/31.jpg)
Future workKnowledge
tracingSMART Andes Today
Design Implementation Calibration Validity Sensitivity
![Page 32: Assessment embedded in step- based tutors (SBTs) CPI 494 Feb 12, 2009 Kurt VanLehn ASU](https://reader035.vdocuments.us/reader035/viewer/2022081603/5697bfd61a28abf838cadcf6/html5/thumbnails/32.jpg)
Summary
An introduction to a common type of ITS– Steps replace answers as main student response
A synthesis of several Bayesian assessment methods– Many knowledge components– Probability of mastery of each knowledge component– One (or more) knowledge components per step– Can use step response patterns to infer mastery– Global traits e.g., meta-cognitive strategy