Download - Parameterized Exercises in Java Programming: using Knowledge Structure for Performance Prediction
Parameterized Exercises in Java Programming: Using Knowledge
Structure for Performance Prediction
Shaghayegh Sahebi (Sherry)1, Yun Huang1, and Peter Brusilovsky1,2
1 Intelligent Systems Program, University of Pittsburgh2 School of Information Sciences, University of Pittsburgh
Parameterized Exercises in Java Programming 2Shaghayegh Sahebi (Sherry)
QuizJET
• Programming questions– Java problems
• Can be designed with parameterized exercises– One question with multiple parameter sets– Can be repeated multiple times by one student
• Authoring tool for Java questions– Create and modify questions– Indexing service to define concepts inside the question
Parameterized Exercises in Java Programming 3Shaghayegh Sahebi (Sherry)
QuizJET: Sample Question
Each question is generated from a template, and students can try multiple attempts.
Students give values for specified variable, or give the output of the code.
A question for practicing skill nested loops
Parameterized Exercises in Java Programming 4Shaghayegh Sahebi (Sherry)
QuizJET: Sample Parameterized Question
Parameterized Exercises in Java Programming 6Shaghayegh Sahebi (Sherry)
Aggregate (MasteryGrids
Services)
Aggregate
UM2
Other (content specific)
PAWSUM Services
Content apps
Server side apps(Apache Tomcat)
Databases (MySQL)
Client interfaceMasteryGrids Interface
Content popout iframe
QuizJet
WebEx
SQLKnot
(a)GUI calls MG
services
direct link
services calls
(b)Aggregate uses
UM services
cbum login
Overall view of the architectureMastery Grids
Parameterized Exercises in Java Programming 8Shaghayegh Sahebi (Sherry)
Mastery Grids
• Students can choose what question to solve– Using social navigation support
• Adding guidance to the question – Use the whole set of data to develop personalized
guidance– Predict how likely the problem will be solved– Avoid too simple and too complex problems
Parameterized Exercises in Java Programming 9Shaghayegh Sahebi (Sherry)
Predicting Students’ Performance (PSP) in Parameterized Questions
• Predicting the student’s capability to perform an educational task
• Assumption: the student can learn by practicing over time by repeating– Time sequence modeling effect on PSP • Will present at the Problem Solving & Strategies session
on Monday
– Knowledge structure effect on PSP• Today’s talk
Parameterized Exercises in Java Programming 10Shaghayegh Sahebi (Sherry)
Traditional PSP: Ignoring Multidimensionality of the Data
• Questions related to topics, concepts, or skills – many dimensions in the data– Structure in the data (knowledge structure)
• Traditional methods: mostly consider student’s past performance– Only consider correct/incorrect attempts of students
(ignoring the multidimensionality of the data) – Bayesian Knowledge Tracing (BKT)– Performance Factor Analysis (PFA)
Parameterized Exercises in Java Programming 11Shaghayegh Sahebi (Sherry)
New Approaches to PSP: Including Multidimensionality of Data
• Considering knowledge structure in PSP– Feature-Aware Knowledge Tracing (FAST) [González-Brenes
et al., ‘13]
– Our suggestion: Tensor Factorization Methods
Parameterized Exercises in Java Programming 12Shaghayegh Sahebi (Sherry)
Our Goal
• Study the effect of knowledge structure modeling in PSP for parameterized questions
• Compare five approaches:– Bayesian Knowledge Tracing (BKT)– Performance Factor Analysis (PFA)– Feature-Aware Knowledge Tracing (FAST)– 3D and 4D Tensor Factorization (3D-BPTF, 4D-
BPTF)
Parameterized Exercises in Java Programming 13Shaghayegh Sahebi (Sherry)
BKT
• Markov Model with two states
• No knowledge structure: Only one type of knowledge component
• Guess, slip, learning, and initial knowledge parameters
Knowledge Structure
Parameterized Exercises in Java Programming 14Shaghayegh Sahebi (Sherry)
PFA
• Regression model
• No knowledge structure
Knowledge Structure
Parameterized Exercises in Java Programming 15Shaghayegh Sahebi (Sherry)
• Extension of BKT• Can include knowledge structure as regression
variables
Knowledge Structure
FAST
Parameterized Exercises in Java Programming 16Shaghayegh Sahebi (Sherry)
Tensor Factorization
• Tensors: n-dimensional arrays
• Used in collaborative-filtering recommender systems– Estimates each tensor as the sum of multiple rank-1
tensors
• Can be extended to as many dimensions – Can include the data structure– Each dimension of the data ≈ one dimension of the
tensor
Parameterized Exercises in Java Programming 17Shaghayegh Sahebi (Sherry)
3D-Tensor Factorization (3D-BPTF)
• Used successfully in PSP for traditional PSP• No knowledge structure• We use Bayesian Probabilistic Tensor
Factorization Model (3D-BPTF) [Xiong et al., 2010]
Stud
ents
Time
Questions/ topics
…
Knowledge Structure
Parameterized Exercises in Java Programming 18Shaghayegh Sahebi (Sherry)
4D-Tensor Factorization (4D-BPTF)
• Used for the first time in PSP• Adds knowledge structure modeling• Can be extended to more dimensions if
needed
Stud
ents
Topics
QuestionsSt
uden
tsTo
pics
Questions
Stud
ents
Topics
Questions
Attempt 1 Attempt 2 Attempt 3
Knowledge Structure
Parameterized Exercises in Java Programming 19Shaghayegh Sahebi (Sherry)
Data
• From QuizJET system• Java programming questions• Six semesters• 166 students• 103 questions• 69.04% majority class (successes)
Parameterized Exercises in Java Programming 20Shaghayegh Sahebi (Sherry)
Study Setup
• Here, each topic can have multiple questions and each question is related to one topic– Two dimensions: questions and topics
• Study 1: traditional approach – Question as knowledge unit
• Study 2: considering knowledge structure– Topic added as knowledge unit
• 5-Fold cross validation– 80% of students in train data, rest in test data– User-stratified
Parameterized Exercises in Java Programming 21Shaghayegh Sahebi (Sherry)
Study 1: Question as knowledge unit
Study 1: comparing traditional approaches(no knowledge structure)
Parameterized Exercises in Java Programming 22Shaghayegh Sahebi (Sherry)
Accuracy of all models: very close to each other
FAST with no additional
parameters
BKT PFA 3D-BPTF73
73.2
73.4
73.6
73.8
74
74.2
74.4
74.6
74.8
Accuracy of Traditional Models
Study 1: comparing traditional approaches(no knowledge structure)
Parameterized Exercises in Java Programming 23Shaghayegh Sahebi (Sherry)
BKT overestimates student’s performance
FAST with no additional
parameters
BKT PFA 3D-BPTF0
200
400
600
800
1000
1200
1400
False Positive
Study 1: comparing traditional approaches(no knowledge structure)
Parameterized Exercises in Java Programming 24Shaghayegh Sahebi (Sherry)
FAST underestimates student’s performance
FAST with no additional
parameters
BKT PFA 3D-BPTF0
100
200
300
400
500
600
700
800
False Negative
Study 1: comparing traditional approaches(no knowledge structure)
Parameterized Exercises in Java Programming 25Shaghayegh Sahebi (Sherry)
FAST Predicts Success Better
FAST with no addi-
tional pa-rameters
BKT PFA 3D-BPTF0
10
20
30
40
50
60
70
80
90
Majority PrecisionMinority Precision
BKT predicts failure better
Study 1: comparing traditional approaches(no knowledge structure)
Parameterized Exercises in Java Programming 26Shaghayegh Sahebi (Sherry)
Study 2: Topic as additional knowledge unit
Study 2: comparing approaches including knowledge structure
Parameterized Exercises in Java Programming 27Shaghayegh Sahebi (Sherry)
FAST and 4D-BPTF more accurate than other approaches
FAST 4D-BPTF BKT PFA 3D-BPTF66
67
68
69
70
71
72
73
74
75
76
Accuracy of Approaches with Additional Knowledge Structure
Study 2: comparing approaches including knowledge structure
Parameterized Exercises in Java Programming 28Shaghayegh Sahebi (Sherry)
3D-BPTF overestimates students’ performance
FAST 4D-BPTF BKT PFA 3D-BPTF0
200
400
600
800
1000
1200
1400
1600
1800
False Positive
Study 2: comparing approaches including knowledge structure
Parameterized Exercises in Java Programming 29Shaghayegh Sahebi (Sherry)
4D-BPTF underestimates students’ performance
FAST 4D-BPTF BKT PFA 3D-BPTF0
100
200
300
400
500
600
False Negative
Study 2: comparing approaches including knowledge structure
Parameterized Exercises in Java Programming 30Shaghayegh Sahebi (Sherry)
4D-BPTF and FAST Predict Success Better
FAST 4D-BPTF BKT PFA 3D-BPTF0
10
20
30
40
50
60
70
80
90
100
Majority PrecisionMinority Precision
Study 2: comparing approaches including knowledge structure
3D-BPTF predicts failure better
Parameterized Exercises in Java Programming 31Shaghayegh Sahebi (Sherry)
FAST and BPTF improved using knowledge structure
FAST 4D-BPTF BKT PFA 3D-BPTF66
67
68
69
70
71
72
73
74
75
76
Question as KC (No Structure)Topic as KC (with Question Structure)
3D-BPTF
Accu
racy
Parameterized Exercises in Java Programming 32Shaghayegh Sahebi (Sherry)
Traditional models declined with topic as KC
FAST 4D-BPTF BKT PFA 3D-BPTF66
67
68
69
70
71
72
73
74
75
76
Question as KC (No Structure)Topic as KC (with Question Structure)Ac
cura
cy
Parameterized Exercises in Java Programming 33Shaghayegh Sahebi (Sherry)
Conclusions
• Accuracy in predicting students performance depends on the input of the method– When ignoring the topic of questions as KCs, all
models perform similarly– When including topic information, in addition to
the question information, the methods that can leverage it perform better
Parameterized Exercises in Java Programming 34Shaghayegh Sahebi (Sherry)
Conclusions
• Adding the extra topic data in the methods that cannot model this information decreases the method’s accuracy
• Knowledge structure can add to the accuracy of PSP in parameterized questions
• Tensor factorization methods are as good, or better than the pioneers PSP methods
Parameterized Exercises in Java Programming 35Shaghayegh Sahebi (Sherry)
Future Work
• Include additional structure into tensor factorization using more dimensions
• Use of other collaborative filtering methods for PSP
• Test on other programming courses
Parameterized Exercises in Java Programming 37Shaghayegh Sahebi (Sherry)
Implementation
• EM algorithm for BKT and set the initial parameters as follows: p(L0) = 0:5 , p(G) = 0:2 , p(S) = 0:1 , p(T) = 0:3 . For running PFA, we use
• the implementation of logistic regression in WEKA [3]. For BPTF and BPMF,
• we utilize the Matlab code prepared by Xiong et. al. We experimented with different latent space dimensions for BPTF and BPMF (5, 10, 20 and 30) and chose the best one, which has the latent space dimension of 10
Parameterized Exercises in Java Programming 38Shaghayegh Sahebi (Sherry)
Matrix Factorization (BPMF)
• From collaborative filtering • Used successfully in PSP for static questions• No attempt sequence modeling• We use Bayesian Probabilistic Matrix
Factorization (BPMF)
1 0 0 01 1 0 10 0 1 10 0 0 1St
uden
ts
Questions/ topics
0.9
0
1.5
0.4
0 1.4
0 0.9
Stud
ents
KCs
0.8
0.5
0 0.3
0 0 0.5
0.8
KCs
Questions/ topics
Parameterized Exercises in Java Programming 40Shaghayegh Sahebi (Sherry)
Study 1: Question as knowledge unit
Accuracy of all models is very close to each other
Parameterized Exercises in Java Programming 41Shaghayegh Sahebi (Sherry)
Study 1: Question as knowledge unit
BKT over estimates the student’s performance
Parameterized Exercises in Java Programming 42Shaghayegh Sahebi (Sherry)
Study 1: Question as knowledge unit
FAST tends to predict more failures for the students
Parameterized Exercises in Java Programming 43Shaghayegh Sahebi (Sherry)
Study 1: Question as knowledge unit
If FAST predicts a success for a student and if BKT predicts a failure for students, their prediction is more likely to be true compared to the other methods.
Parameterized Exercises in Java Programming 44Shaghayegh Sahebi (Sherry)
Study 2: Topic as knowledge unit
FAST and 4D-BPTF perform significantly better than all other approaches
Parameterized Exercises in Java Programming 45Shaghayegh Sahebi (Sherry)
Study 2: Topic as knowledge unit
BKT and PFA perform similarly to their results in Study 1 and 3D-BPTF on topics is slightly weaker than 3D-BPTF on questions in terms of accuracy.
Parameterized Exercises in Java Programming 46Shaghayegh Sahebi (Sherry)
Mastery Grids
• Visual, interactive, adaptive E-learning platform– Multi-facet social comparison– Multi-type learning materials support– Social navigation– Personalized guidance
• Integration with other systems with little set up and modification