1. Cristina Conati Department of Computer Science University of British Columbia Beyond problem solving: student adaptive support for open-learning environments 2. Intelligent Tutoring Systems (ITS) Create computer-based tools that support individual learners by autonomously and intelligently adapting to their specific needs Student Model Tutor Domain Model Adaptive Interventions 3. Adaptation Cycle Student Model Student states/traits modeled Goals Knowledge Behavioral Regularities Preferences Emotions Cognitive Load 4. Adaptation Cycle User Model Interface Actions Gestures Gaze Patterns Body Movements Goals Knowledge Behavioral Regularities Preferences Emotions Cognitive Load Input sources User states/traits modeled 5. Adaptation Cycle User Model Input Interface Actions Gestures Gaze Patterns Body Movements Goals Knowledge Behavioral Regularities Preferences Emotions Cognitive Load Give adaptive hints/feedback on task Adapt curriculum Recommend activities Tailor Information Presentation . Forms of adaptation Input sources User states/traits modeled 6. Achievements In the last 20 years, there have been many successful initiatives in devising Intelligent Tutoring Systems (e,g, Woolf 2009, Building Intelligent Interactive Tutors, Morgan Kaufman) Mainly ITS that provide individualized support to problem solving activities Well defined problem solutions => guidance on problem solving steps Clear definition of correctness => basis for feedback 7. Beyond ILEs for Problem Solving Problem solving is a very important component of learning Other forms of instruction, however, can contribute to learning At different stages of the learning process For learners with specific needs and preferences 8. Beyond Coached Problem Solving We have been investigating how to provide adaptive support for some of these activities Learning from examples (e.g., Conati Muldner Carenini, TICL 2005, Muldner and Conati, JAIED 2010) Playing educational games (e.g, Conati and Zhao IUI 2004, Conati and Maclaren UMUAI 2009, Muir and Conati ITS 2012), Conati et al., IJAIED 2014 Exploring interactive simulations (e.g, Bunt and Conati, UMUAI 2002, Conati and Merten KBS 2007, Kardan and Conati EDM 2011, UMAP 2012, 2013) 9. Interactive Simulations for Learning Support active learning [e.g.,Van Joolingen et al., 2007] Allow for exploratory interaction Expand student understanding of theoretical concepts via experimentation with concrete examples Can be very effective tools for self-study / MOOCS 10. AISpace (Amershi et al., 2007) Suite of interactive simulations of common Artificial Intelligence algorithms Used regularly in our AI courses Google AISpace if you want to try it out CSP (Constraint Satisfaction Problems) Applet visualizes the working of the AC3 algorithm An Example 11. AISpace CSP Applet 12. Issue with Simulations Not all students learn well from exploratory activities Important to provide adaptive support for those students who need help. But.. How to model what the student should do? How to provide adaptive support that fosters learning ? maintains student initiative and engagement? 13. Student Modeling Challenges Activities more open-ended and less well-defined than pure problem solving No clear definition of behavior correctness No clear definition of behavior effectiveness. Hard to model the necessary student behaviors and related skills 14. Our Approach Learner models and adaptive hints based on behaviors discovered via data mining 15. Behavior Discovery Via Data Mining Association Rules Mining Clustering Actions Logs Other Data Fe atu re Ve cto rs Vector of Interaction Features - Frequency Of Actions - Latency Between Actions Extract rules describing distinguishing patterns in each cluster Groups together students that have similar interaction behaviors Interpret in terms of learning Experts Performance Measure(s) (Amershi and Conati, 2009, Kardan and Conati, 2011, 2013) 16. 17 Test Bed - CSP Applet 17. User Study (Kardan and Conati 2011) 64 subjects 13,078 actions More than 17 hours of interaction Study Material on AC3 Pretest Two CS problems with the CSP Applet Posttest 18. Found 2 clusters Statistically significant difference in Learning Gains High Learners (HL) and Low Learners (LL) clusters Feature vectors Clustering Behavior Discovery Rule MiningClustering Features: frequencies of use for each action pause duration between actions (Mean and SD) 7 actions 21 features 19. Usefulness: Sample Rules HL members: Use Direct Arc Click action very frequently (R1). HL cluster: R1: Direct Arc Click frequency = Highest (Conf =100%, Class Cov = 100%) LL cluster: R2: Direct Arc Click Pause Avg = Lowest (Conf =100%, Class Cov = 100%) R3: Direct Arc Click frequency = Lowest (Conf = 93%, Class Cov=93.5%) LL members: Use Direct Arc Click sparsely (R3) Leave little time between a Direct Arc Click and the next action (R2) Feature vectors Clustering Behavior Discovery Rule Mining 20. Great, but what do we do with this? Use the learned clusters and rules to classify a new student based on her behaviors Use detected behaviours for adaptive support Promoting the behaviours conducive to learning Discouraging/preventing detrimental behaviours 21. The User Modeling Framework 2 2 Association Rules Mining Clustering Feature Vector Calculation Online Classifier Adaptive Interventions Behavior Discovery User Classification Actions Logs Other Data F e at u re New users Actions Vector of Interaction Features If user is a LL and uses Direct Arc Click very infrequently (R3) Then prompt this action If user is a LL and pauses very briefly after a Direct Arc Click (R2) Then take action to slow her down 22. Classifier Evaluation on CSP Applet Data (Kardan and Conati 2011) Accuracy as a function of observed actions Based on leave-one-out cross validation 23. Challenges Not all students learn well from exploratory activities Important to provide adaptive support for those students who need help. But.. How to model what the student should do? How to provide adaptive support that fosters learning ? maintains student initiative and engagement? 24. Adaptive Incremental Hints: Level 1 (Kardan and Conati, 2013, 2105) Classifier User Model detects a Low Learner that Uses Direct Arc Click sparsely (R3) 25. Adaptive Incremental Hints: level 2 26. Evaluation Controlled user study : 18 users studied 3 CSP problems with the adaptive CSP applet 18 users studied the same problems with the original CSP Applet Study Material on AC3 Pretest Three CS problems with adaptive or original CSPApplet Posttest similar design used previously for data collection 27. Results Original CSP applet Adaptive CSP applet Learning Gains Students working with the adaptive CSP applet learned significantly more (p < 0.05) 28. Learning Gain: PreTestCondition 29. Results: Acceptance of Interventions 30. Current Work Apply our framework to a more complex interactive simulation : PhET DC Circuit Construction Kit (CCK) 31. Interaction Demo 32. Complex Interaction 22 components, eg: Basic circuit elements Measurement tools 25 actions, eg: Circuit (components) Add Move Remove Join Measurement Voltage Current Interface Simulation settings Window 6 outcomes, eg: Light intensity change Current change Fire Measurement Reading change Large variety of ways to interact See Conati et al, AIED 2015 for more details on how the dealt with the complexity 33. Building Representations to Capture Complex Interaction Layered representation for interaction behaviors 4 layers: Components, Actions, Outcomes (from the logs) Action Families (e.g., building, revising, testing,) Represent complex interaction-events, eg: Current_change.Revise.join.wire Student generated a current change while revising the circuit by joining two wires See Conati et al, AIED 2015 for more details 34. The User Modeling Framework 3 6 Association Rules Mining Clustering Feature Vector Calculation Online Classifier Adaptive Interventions Behavior Discovery User Classification Actions Logs Other Data F e at u re New users Actions Vector of Interaction Features Testing our framework on CCK 35. Testing the framework on CCK Data collected from a lab study with CKK 96 first year physics students All students were given a general learning goal: Explore how resistors affect the behavior of circuits by exploring different combinations of resistors and resistances ~ 25 minutes of interaction, 1000 actions per student Collected pre and post test data 36. The User Modeling Framework 3 8 Association Rules Mining Clustering Feature Vector Calculation Online Classifier Adaptive Interventions Behavior Discovery User Classification Actions Logs Other Data F e at u re New users Actions Vector of Interaction Features 37. Clustering on Interaction Behaviors #Members Average Post-test Score Average Pre- test Score p-value Effect Size (partial eta squared) High 61 .609 .465 .013 .065 Low 35 .511 .470 Identified two clusters of users with significantly different learning gains 38. The User Modeling Framework 4 0 Association Rules Mining Clustering Feature Vector Calculation Online Classifier Adaptive Interventions Behavior Discovery User Classification Actions Logs Other Data F e at u re New users Actions Vector of Interaction Features 39. Classification Accuracy Overtime 40. The User Modeling Framework 4 2 Association Rules Mining Clustering Feature Vector Calculation Online Classifier Adaptive Interventions Behavior Discovery User Classification Actions Logs Other Data F e at u re New users Actions Vector of Interaction Features 41. Generated Association Rules Identified 15 behavior patterns intuitively associated with learning, e.g. that it is productive To test infrequently To frequently change resistance of resistors Limit the usage of light bulbs and changes to their light intensity 42. Conclusions User modeling framework for providing personalized support to learning with interactive simulations Relevant behaviours are discovered via data mining Very encouraging results with CSP applet Identified clusters with behaviors related to learning or lack thereof Online classifier: good accuracy, soon enough to generate adaptive interventions Adaptive interventions can be derived automatically from the detected behaviors (Kardan and Conati, 2012, 2013) Evidence that they can help learning! (Karnan and Conati 2015) Also initial evidence that the approach transfers to a more complex simulation (Conati al 2015) 43. Future Work Build adaptive interventions for CCK simulation Apply the approach to more simulations Let us know if you have any that you would like to try!