Slide 1

CurriM: Curriculum Mining Mykola Pechenizkiy TU Eindhoven Learning Analytics Innovation 10 October 2012 SURFfoundation, Utrecht, the Netherlands

Slide 2

Initial Motivation for CurriM Current practice: We think we know what our curriculum is and how the students study. But is this true? CurriM aims at providing tools to analyze how the students actually study Who would benefit from our tool? Directors of education, study advisers, students Goal: showcase the potential and feasibility Data mining and process mining techniques 10 years of TUE administrative data; exam grades Learning Analytics @Surf 10 October 2012, Utrecht, 1CurriM: Curriculum Mining Mykola Pechenizkiy, Eindhoven University of Technology

Slide 3

Questions for CurriM to Answer What is the real academic curriculum (study program)? How do students really study? Is there a typical (or the best) way to study? Do current prerequisites make sense? Is the particular curriculum constraint obeyed? How likely is it that a student will finish the studies successfully or will drop out? What is my expected time to finish? Should I now take courses A & B & C or C & D? Learning Analytics @Surf 10 October 2012, Utrecht, 2CurriM: Curriculum Mining Mykola Pechenizkiy, Eindhoven University of Technology

Slide 4

Refocused to Target Students as Users Awareness tool supporting interactive querying: How does a course relate to the program? Prerequisites, follow up dependencies How am I doing wrt the averages, top 10%? Aggregates/OLAP What is my expected time to finish? Predictive modeling Should I now take courses A & B & C or C & D? Collaborative filtering style recommendations (based on the received feedback) Learning Analytics @Surf 10 October 2012, Utrecht, 3CurriM: Curriculum Mining Mykola Pechenizkiy, Eindhoven University of Technology

Slide 5

CurriM UI Demo Learning Analytics @Surf 10 October 2012, Utrecht, 4CurriM: Curriculum Mining Mykola Pechenizkiy, Eindhoven University of Technology

Slide 6

Where is EDM/LA? Curriculum model: Codified constraints with Colored Petri net and LTL Prerequisites, follow up dependencies, 3 out of 5 selection, number of attempts, mandatory courses etc. Input: domain knowledge and output of patters mining Awareness and automated conformance checking Is the currently chosen path compliant with the official guidelines and follows data driven recommendations Computed aggregates and mined pattern from the data Data driven recommendations and predictions What is my expected time to finish? Should I take now courses A & B & C or C & D? (hidden from the users behind GUI) Learning Analytics @Surf 10 October 2012, Utrecht, 5CurriM: Curriculum Mining Mykola Pechenizkiy, Eindhoven University of Technology

Slide 7

Main Results Software prototype CurriM as ProM plugin, Focus on GUI + architecture/interfaces Demonstrates the concept Experiments with TUE dataset Prerequisites, bottleneck/predictive courses Recommendations Data quality is the key Clear motivation and need for a continuation The concept is found to be promising Potential and feasibility is shown Roadmap Learning Analytics @Surf 10 October 2012, Utrecht, 6CurriM: Curriculum Mining Mykola Pechenizkiy, Eindhoven University of Technology

Slide 8

Why Do Students Like the Concept? CurriM is a tool that Provides orientation: Curriculum as a guide and motivation See the connections and dependencies Provides awareness and recommendations Global: how good is their personal education route, where they currently are, where they are heading, how well they do in comparison with others Local: what would it mean to take course X Enables better planning and regular monitoring Focus on what looks important, not just interesting Learning Analytics @Surf 10 October 2012, Utrecht, 7CurriM: Curriculum Mining Mykola Pechenizkiy, Eindhoven University of Technology

Slide 9

Main Lessons Learnt Data quality is the key Administrative DBs and existing data collection organization do not keep EDM/LA in mind Lots of preprocessing and reorganization is required Meta-data is the other key (lacking codifiability) Everything that is scattered in study guides and minds of study advisors should become easy to codify Curriculum changes more often than we tend to think Semesters-trimesters-quartiles, courses & course ids Being flexible (written vs. unwritten rules) too much Effectively means no formal curriculum Learning Analytics @Surf 10 October 2012, Utrecht, 8CurriM: Curriculum Mining Mykola Pechenizkiy, Eindhoven University of Technology

Slide 10

Conclusions CurriM can become a big success The students seem to like the idea It is promising and it is feasible; but it is a long way from the current concept to a fully functional and usable tool Surf funding opportunity in LA was nice Triggered us to take concrete practical steps, a tool rather than techniques development; But a more serious commitment is needed to make a real breakthrough and bring CurriM into the educational practice Learning Analytics @Surf 10 October 2012, Utrecht, 9CurriM: Curriculum Mining Mykola Pechenizkiy, Eindhoven University of Technology

Slide 11

Continuation Roadmap Working out the full cycle of the information flows including pattern mining, predictions and recommendations, and its integration/parallelization with the administrative processes Working out different views and functionality for students vs. educators, HCI/usability aspects Improve data quality collection Facilitate knowledge base construction (meta- data, mappings) Facilitate curriculum formalization for faculties (tooling) Conditioned wrt funding opportunities Learning Analytics @Surf 10 October 2012, Utrecht, 10CurriM: Curriculum Mining Mykola Pechenizkiy, Eindhoven University of Technology

Slide 12

Project Team Project leader: dr. Mykola Pechenizkiy educational data mining expert Driving force: Pedro Toledo software developer, applied researcher Technology experts: Prof. dr. Paul De Bra Human-computer interaction and databases expert dr. Toon Calders pattern mining expert, assistant professor dr. Nikola Trcka collaborator on curriculum mining, postdoc dr. Boudewijn van Dongen process mining expert, assistant professor dr. Eric Verbeek ProM software expert, scientific programmerProM software Domain experts Several domain experts, i.e. responsible educators, are available for CurriM on request: dr. Karen Ali (STU), Prof. dr. Mark de Berg (CSE) Learning Analytics @Surf 10 October 2012, Utrecht, 11CurriM: Curriculum Mining Mykola Pechenizkiy, Eindhoven University of Technology

Slide 13

Additional slides Including some from the original proposal Learning Analytics @Surf 29 February 2012, Utrecht, 12CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 14

Execution plan Task 1. Developing the first software prototype for academic curriculum modeling. As mini R&D cycles : identifying types of curriculum specific patterns we need to mine from the event logs (in collaboration with the domain experts) and to include in the curriculum modeling and developing corresponding pattern mining and pattern assembling techniques; Implementing techniques and integrating it with ProM that provides an important process mining foundation framework and many of the building blocks for curriculum modeling software; testing a particular piece of software. Learning Analytics @Surf 29 February2012, Utrecht, 13CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 15

Execution plan Task 2. Case study: modeling the curriculum of the Department of Computer Science, TUE; Goals: Validating the correctness and usefulness (to the end users, i.e. teachers, study advisers, students) of the developed curriculum mining techniques and their implementations. Developing guidelines for managing the curriculum related data to avoid the problems we will encounter or envision during the case study. Task 1 and Task 2 will run simultaneously ensuring timely feedback. Learning Analytics @Surf 29 February2012, Utrecht, 14CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 16

Execution plan Task 3. Creating a roadmap for further study and development of the curriculum modeling toolset Develop R&D agenda for the coming years. This includes identification of not only research challenges i.e. answering the question what kind of new data mining and process mining techniques are needed to address the peculiarities of the curriculum mining domain? but also the strategy of the smooth technology transfer to the prospective end users, i.e. early adopters (e.g. TUE or 3TU departments) that would help to validate the usability and usefulness of the curriculum mining software in the wild. Learning Analytics @Surf 29 February2012, Utrecht, 15CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 17

Project Team Task 3. Creating a roadmap for further study and development of the curriculum modeling toolset Develop R&D agenda for the coming years. This includes identification of not only research challenges i.e. answering the question what kind of new data mining and process mining techniques are needed to address the peculiarities of the curriculum mining domain? but also the strategy of the smooth technology transfer to the prospective end users, i.e. early adopters (e.g. TUE or 3TU departments) that would help to validate the usability and usefulness of the curriculum mining software in the wild. Learning Analytics @Surf 29 February2012, Utrecht, 16CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 18

Learning Analytics Seminar, August 30-31, Utrecht, NL 17Educational Data Mining & Learning Analytics for All: Potential, Dangers, Challenges Mykola Pechenizkiy, Eindhoven University of Technology

Slide 19

Educational Process Mining Toolbox Learning Analytics @Surf 29 February 2012, Utrecht, 18CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 20

Intuition suggests that curriculum is Structured and easy to understand as we think there are not that many options to choose from It may look just like this one: but the data may suggest that it looks different Learning Analytics @Surf 29 February 2012, Utrecht, 19CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 21

data may suggest that students show somewhat more diverse behaviour: Learning Analytics @Surf 29 February2012, Utrecht, 20CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 22

How will students and teachers benefit? Our intention is to make the curriculum- related patterns and models accessible not only to the study advisors or directors of education but to everyone involved in the educational process including lecturers and students. Thus students will become more aware that e.g. studying well for the Logic course will help them to go smoothly through studying the basic of Databases and thus avoid further delays in the curriculum Learning Analytics @Surf 29 February2012, Utrecht, 21CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 23

Process Analysis/Conformance Checking ex. Learning Analytics @Surf 29 February 2012, Utrecht, 22CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 24

Two Different Tasks Isolate a set of standard curriculum patterns and based on these patterns mine the curriculum as an executable quantified formal model and analyze it, or first (manually) devise a formal model of the assumed curriculum and test it against the data. Event Log - MXML format supported by ProM Typical forms of requirements in the curriculum Colored Petri net Learning Analytics @Surf 29 February 2012, Utrecht, 23CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 25

Application Scenarios Student TimestampEvents AS12, 3, 5 AS26, 1 AS31 BS14, 5, 6 BS32 BS47, 8, 1, 2 BS51, 6 CS11, 8, 7 Scenario 1: Find most common types of behavior (and cluster them) Scenario 2: Find emerging patterns: such patterns, which capture significant differences in behavior of students who graduated vs. those students who did not changes in behaviour of students from year 2006-07 to 2007-08. in both cases we search for such patters which supports increase significantly from one dataset to another (i.e. in space in the first case and in time in the second case) Scenario 3: After finding a bottleneck, find frequent patterns that describe it, i.e. for which students it is the bottleneck and why Learning Analytics @Surf 29 February 2012, Utrecht, 24CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 26

Example 2-out-of-3 Pattern Check At least 2 courses from { 2Y420,2F725,2IH20 } must be taken before graduation : An higher level abstraction can be developed on a longer run to avoid we aim at developing a Learning Analytics @Surf 29 February 2012, Utrecht, 25CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 27

Process Discovery Example Learning Analytics @Surf 29 February 2012, Utrecht, 26CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 28

Which Courses Are Difficult/Easy for Which Students? Learning Analytics @Surf 29 February 2012, Utrecht, 27CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 29

Many more examples of Visual Analytics from MagnaView Students Performance Started in the Same Year Learning Analytics @Surf 29 February 2012, Utrecht, 28CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 30

References Trka, N., Pechenizkiy, M. & van der Aalst, W. (2010) "Process Mining from Educational Data (Chapter 9)", In Handbook of Educational Data Mining., pp. 123-142. London: CRC Press. Pechenizkiy, M., Trka, N., Vasilyeva, E., van der Aalst, W. & De Bra, P. (2009) Process Mining Online Assessment Data, In Proceedings of 2nd International Conference on Educational Data Mining (EDM'09), pp. 279-288. Trka, N. & Pechenizkiy, M. (2009) From Local Patterns to Global Models: Towards Domain Driven Educational Process Mining, In Proceedings of Ninth International Conference on Intelligent Systems Design and Applications (ISDA'09), pp. 1114-1119. Bose, R.P.J.C., van der Aalst, W.M.P., Zliobaite, I. & Pechenizkiy, M. (2011) Handling Concept Drift in Process Mining, In Proceedings of 23rd International Conference on Advanced Information Systems Engineering CAiSE'2011, Lecture Notes in Computer Science 6741, Springer, pp. 391-405. Dekker, G., Pechenizkiy, M. & Vleeshouwers, J. (2009) Predicting Students Drop Out: a Case Study, In Proceedings of the 2nd International Conference on Educational Data Mining (EDM'09), pp. 41-50. http://www.processmining.org/ Learning Analytics @Surf 29 Febnuary 2012, Utrecht, 29CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology

Slide 31

Short CV of the Project Leader Mykola Pechenizkiy Assistant Professor at Dept. of Computer Science, TU/e Research interests: data mining and knowledge discovery; Particularly predictive analytics for information systems serving industry, commerse, medicine and education. http://www.win.tue.nl/~mpechen/http://www.win.tue.nl/~mpechen/ - projects, pubs, talks etc. Major recent EDM-related activities:

Slide 32

Confirmed interest in CurriM at TUE Dr. Karen S. Ali - Director of Education and Student Service Center, STU Prof. Dr. Mark de Berg - Director of the graduate program, Dept. of Computer Science Dr. Marloes van Lierop - Director of the bachelor program, Dept. of Computer Science Study advisers at different faculties Learning Analytics @Surf 29 February 2012, Utrecht, 31CurriM: Curriculum Mining Project Proposal Mykola Pechenizkiy, Eindhoven University of Technology