a framework for automatic empirically-based metadata generation intelligent learning object guide...

A Framework for Automatic Empirically-Based Metadata Generation Intelligent Learning Object Guide (iLOG) S.A. Riley a, L.D. Miller a, L.-K. Soh a, A. Samal a, and G. Nugent b a University of NebraskaLincoln: Department of Computer Science and Engineering b University of NebraskaLincoln: Center for Research on Children, Youth, Families and Schools

Overview Intelligent Learning Object Guide 2 Introduction: What is a Learning Object (LO)? Why do we need LO metadata? Metadata problems and iLOG solution iLOG Framework LO Wrapper MetaGen (metadata generator) Data Logging Data Extraction Data Analysis (feature selection, rule mining, statistics) Conclusions and Future Work

Introduction: What is a learning object? Intelligent Learning Object Guide 3 Self-contained learning content Ideally, each covers a single topic Serve as building blocks for lessons, modules, or courses Can be reused in multiple instructional contexts Learning Object LO Metadata iLOG Learning Object structure: Content: tutorial, exercises, assessment Metadata

Introduction: What is a learning object? Intelligent Learning Object Guide 4 Tutorial Assessment Exercises The iLOG LOs contain a tutorial, exercises, and assessment Each covers a bite-sized introductory computer science topic

Intelligent Learning Object Guide 5 Repositories for LOs are being constructed However, there are barriers to effective utilization of these repositories: Learning Context: not all LOs, even on the same topic, are suitable for use in a given learning context Uncertainty: we cannot be certain what will happen with real-world usage Search and Retrieval: current metadata is not machine-readable, and thus is not adequate to automate the search for LOs Introduction: Why do we need LO metadata? Learning Object LO Metadata LO Repository Learning Object LO Metadata Learning Object LO Metadata Learning Object LO Metadata Learning Object LO Metadata

Introduction: Why do we need LO metadata? Intelligent Learning Object Guide 6 Learning Context: Students are highly varied: Pre-existing knowledge, cultural background, motivation, self-efficacy, etc. Uncertainty: Cannot be certain what will happen when actual students use an actual LO: Good for students with low self-efficacy Inherent gender bias Bad for students without Calculus experience Search and Retrieval: Metadata is fundamental to an instructors ability to use LOs: Guide in the LO selection process Help prevent the feeling that e-learning is too complicated

Intelligent Learning Object Guide 7 So how do we enable instructors to locate appropriate LOs for their students???

Introduction: Metadata problems and iLOG solution Current MetadataIdeal Metadata Intelligent Learning Object Guide 8 Manual generation by course designer Based only on designer intuition Metadata format inconsistent / incomplete Human- but not machine- readable Automated generation Based on empirical usage Consistent metadata suitable for guiding LO selection Both human and machine- readable Current metadata standards are insufficient (Freisen, 2008) There are ample opportunities for making e-learning more intelligent (Brooks et al., 2006)

Intelligent Learning Object Guide 9 The iLOG solution is: General: iLOG is based on established learning standards We use the SCORM learning object standard, the IEEE LOM metadata standard, and the Blackboard LMS Furthermore, it is compatible with existing LOs and does not require modification to the LOs (noninvasive) The iLOG framework can also be applied to other standards Automatic: iLOG metadata is automatically generated and updated Interpretable: iLOG metadata is both human and machine readable Introduction: Metadata problems and iLOG solution

Intelligent Learning Object Guide 10 LO Wrapper: logs student behaviors when using LO MetaGen : generates empirical usage metadata using data mining techniques Works noninvasively with pre-existing LOs using standard learning management systems (LMSs) Introduction: Metadata problems and iLOG solution LO Wrapper Learning Object LO Metadata LO Wrapper Learning Object LO Metadata Learning Management System (LMS) LO Wrapper Learning Object LO Metadata

Related Work Intelligent Learning Object Guide 11 Automatic metadata generation Primarily focuses on content taxonomies (Roy et al., 2008; Jovanovic et al., 2006) Mining student behavior log files Mining has been shown to have a positive impact on instruction and learning (Kobsa et al., 2007) Standardization of educational log file data Significant progress has been made with tutor-message format standard (PSLC DataShop)

iLOG Framework Intelligent Learning Object Guide 13 iLOG dataset Log Files and Existing Metadata Data- base Rule Mining Feature Subset Feature Selection Statistics Generation MetaGen Rules and Statistics LO Wrapper LO Metadata Learning Object Two components: LO Wrapper and MetaGen Data Analysis Data Extraction Data Logging

iLOG Framework: LO wrapper Intelligent Learning Object Guide 14 LO Wrapper: Wraps around an existing LO Intercepts student interactions and logs them to a database Does not require changing the LO LO Wrapper Learning Object LO Metadata

iLOG Framework: MetaGen Intelligent Learning Object Guide 15 iLOG dataset Data- base Rule Mining Feature Subset Feature Selection Statistics Generation MetaGen Rules and Statistics MetaGen modules: Data Logging, Data Extraction, Data Analysis Data Analysis Data Extraction Data Logging

iLOG Framework: MetaGenLogging Intelligent Learning Object Guide 16 Potential data sources: Interactions: clicks, time spent, etc. Surveys: demographic, motivation, self-efficacy, evaluation Assessment scores Log Files Data- base Data Logging MetaGen LO Wrapper LO Metadata Learning Object

iLOG Framework: MetaGenLogging Intelligent Learning Object Guide 17 Static Learner Data Static LO DataInteraction Data Baseline motivation Baseline self- efficacy Gender Major GPA SAT/ACT score Topic Length Degree of difficulty Level of feedback. Blooms level for assessment questions Total time on tutorial Total time on exercises Total time on assessment Min time spent on a tutorial page Max time spent on a tutorial page Avg. time per assessment question Data sources used in our iLOG deployment:

iLOG Framework: MetaGenExtraction Intelligent Learning Object Guide 18 Data Extraction: Uses Java application to query the relational database and extract a flat dataset suitable for data mining: Student Behaviors: Average time per tutorial page, Total time on assessment, etc. Student Characteristics: Total motivation self-rating, GPA, Gender, etc. iLOG dataset Log Files and Existing Metadata Data- base Data Extraction Data Logging MetaGen LO Wrapper LO Metadata Learning Object

iLOG Framework: MetaGenAnalysis Intelligent Learning Object Guide 19 Data Analysis (feature selection): Uses ensemble of feature selection algorithms Seeks to identify student behaviors and characteristics that are relevant to learning outcomes iLOG dataset Log Files and Existing Metadata Data- base Feature Subset Feature Selection MetaGen LO Wrapper LO Metadata Learning Object Data Analysis Data Extraction Data Logging

iLOG Framework: MetaGenAnalysis Intelligent Learning Object Guide 20 Feature selection (FS) is used to find a subset of variables (features) that is sufficient to describe a dataset (Guyon et al., 2003) Different techniques may generate different results Instead, our goal was to find ALL features relevant to learning outcomes Thus, the feature selection ensemble members vote on which features they identify as most relevant All features FS#1 FS#2 FS#3

iLOG Framework: MetaGenAnalysis Intelligent Learning Object Guide 21 Notable Results: Relevant features varied widely across LOs Discovered unexpected patterns: Possible gender bias, Calculus bias, etc. Logic 2 Searching Attribute Number of Times Selected AttributeNumber of Times Selected highestMath gender takenCalculus assessStdDevSecAboveAvg? wasAnyPartConfusing? 16 13 GPA assessMinSecPageBelowAvg? assessmentMinScondsOnAPage believeLODifficultToUnderstand courseLevel 14 11 10 9

iLOG Framework: MetaGenAnalysis Intelligent Learning Object Guide 22 Rule Mining: Uses Tertius algorithm for predictive rule mining Generates rules from selected features (along with rule strength) iLOG dataset Log Files and Existing Metadata Data- base Rule Mining Feature Subset Feature Selection MetaGen LO Wrapper LO Metadata Learning Object takenCalculus? = no fail (.52) currentTotalMotivationAboveAvg? = no fail (.52) gender = female fail (.36) Data Analysis Data Extraction Data Logging

iLOG Framework: MetaGenAnalysis Intelligent Learning Object Guide 23 Statistics Generation: Empirical data: time to complete, pass/fail rates, and student ratings of LO iLOG dataset Log Files and Existing Metadata Data- base Rule Mining Feature Subset Feature Selection Statistics Generation MetaGen LO Wrapper LO Metadata Learning Object successRate = 51% averageTime = 433 seconds averageStudentRating = 4.3/5.0 Data Analysis Data Extraction Data Logging

iLOG Framework: MetaGenAnalysis Intelligent Learning Object Guide 24 Logic 2Intro CS for non-majors assessmentStdDevSecondsAboveAvg? = yes fail (.35) assessmentMaxSecondsOnAQuestion = high fail (.33) highestMath = precalculus fail (.28) gender = female fail (.24) successRate = 51% averageTime = 433 seconds averageStudentRating = 4.3/5.0 Logic 2--Intro CS for majors baselineStdDevMotivation = low fail (.72) takenCalculus? = no fail (.52) currentTotalMotivationAboveAvg? = no fail (.52) successRate = 38% averageTime = 688 seconds averageStudentRating = 4.16/5.0 Logic 2Honors Intro CS for majors OpinionOfLOUsability = negative fail (.59) BelieveLOAnAidToUnderstanding = yes pass (.49) BelieveLONeedsMoreDetail = yes fail (.43) gender = female fail (.36) successRate = 55% averageTime = 799 seconds averageStudentRating = 3.43/5.0 Appear to be different predictors of success for different learning contexts: Honors: student impression of LO, gender Majors: motivation, math experience Non-majors: long time spent on assessment, math experience, gender

iLOG Framework: MetaGenAnalysis Intelligent Learning Object Guide 25 Logic 2Intro CS for non-majors assessmentStdDevSecondsAboveAvg? = yes fail (.35) assessmentMaxSecondsOnAQuestion = high fail (.33) highestMath = precalculus fail (.28) gender = female fail (.24) successRate = 51% averageTime = 433 seconds averageStudentRating = 4.3/5.0 Logic 2--Intro CS for majors baselineStdDevMotivation = low fail (.72) takenCalculus? = no fail (.52) currentTotalMotivationAboveAvg? = no fail (.52) successRate = 38% averageTime = 688 seconds averageStudentRating = 4.16/5.0 Logic 2Honors Intro CS for majors OpinionOfLOUsability = negative fail (.59) BelieveLOAnAidToUnderstanding = yes pass (.49) BelieveLONeedsMoreDetail = yes fail (.43) gender = female fail (.36) successRate = 55% averageTime = 799 seconds averageStudentRating = 3.43/5.0 Inverse relationship: time spent on LO and student ratings: Advanced students may have higher expectations (lower ratings) Advanced students may care more about the material (time spent)

iLOG Framework: MetaGenAnalysis Intelligent Learning Object Guide 26 Rules and Statistics: Usage statistics and rules are combined to form empirical usage metadata iLOG dataset Log Files and Existing Metadata Data- base Rule Mining Feature Subset Feature Selection Statistics Generation MetaGen LO Wrapper LO Metadata Learning Object Rules and Statistics Data Analysis Data Extraction Data Logging

iLOG Framework: Our Implementation Intelligent Learning Object Guide 27 LO wrapper: HTML document that uses Java-script to record and timestamp student interactions with the LO (e.g., page navigation, clicks on a page, etc.). Uses a modification of the Easy SCO Adapter 1 to interface with the SCORM API and retrieve student assessment results from the LMS. Uses JavaScript to transmit interaction data to MetaGen MetaGen: Data logging: uses PHP to store student interaction data into a MySQL database. Data extraction: uses Java to query the database and process the data into the iLOG dataset. Data analysis: uses the Weka (Witten, 2005) implementations of several feature selection algorithms to generate the iLOG data-subset and the (Flach, 2001) predictive rule mining algorithm to generate empirical usage metadata rules. 1 [http://www.ostyn.com/standards/demos/SCORM/wraps/easyscoadapterdoc.htm#license]http://www.ostyn.com/standards/demos/SCORM/wraps/easyscoadapterdoc.htm#license

Conclusions Intelligent Learning Object Guide 29 iLOG: a framework for automatic, empirical metadata generation: LO Wrapper component: Wraps noninvasively around pre-existing learning objects (LOs) Automatically collects and logs student interaction data Resulting LOs can be played on a standard LMS, such as Blackboard MetaGen component (metadata generator): Uses data mining to create empirical usage metadata: Feature selection: provides insights on which student characteristics and behaviors may contribute to success in different learning contexts. Rule mining: uses salient features to generate rules predicting success Usage statistics: empirical evidence of time to complete, scores, etc. iLOGs empirical use metadata should enable instructors to locate LOs that are appropriate to their students learning context

Future Work: Closing the Loop Intelligent Learning Object Guide 30 iLOG dataset Log Files and Existing Metadata Data- base Rule Mining Feature Subset Feature Selection Statistics Generation MetaGen Rules and Statistics LO Wrapper LO Metadata Learning Object Method to automatically write empirical usage metadata to the LO metadata file Method to integrate new metadata with existing metadata Data Analysis Data Extraction Data Logging

References Intelligent Learning Object Guide 31 IEEE 1484.12.1-2002 Standard for Learning Object Metadata (LOM). Retrieved January 7, 2009, from http://ltsc.ieee.org/wg12/files/LOM_1484_12_1_v1_Final_Draft.pdf N. Friesen, The International Learning Object Metadata Survey. Retrieved August 7, 2008, from http://www.irrodl.org/index.php/irrodl/article/view/195/277/ C. Brooks, J. Greer, E. Melis, C. Ullrich, Combining ITS and eLearning Technologies: Opportunities and Challenges, Proc. 8 th Int. Conf. on Intelligent Tutoring Systems (2006), 278-287. D. Roy, S Sarkar, S. Ghose, Automatic Extraction of Pedagogic Metadata from Learning Content, Int. J. of Artificial Intelligence in Education 18 (2008), 287-314. J. Jovanovic, D. Gasevic, V. Devedzic, Ontology-Based Automatic Annotation of Learning Content, Int. J. on Semantic Web and Information Systems, 2(2) (2006), 91-119. B. Jong, T. Chan, Y. Wu, Learning Log Explorer in E-Learning Diagnosis, IEEE Transactions on Education 50(3) (2007), 216- 228. E. Garcia, C. Romero, S. Ventura, C. Castro, An architecture for making recommendations to courseware authors using association rule mining and collaborative filtering, User Modeling and User-Adaptive Interaction (to appear). E. Kobsa, V. Dimitrova, R. Boyle, Adaptive Feedback Generation to support teachers in web-based distance education, User Modeling and User-Adapted Interaction 17 (2007), 379-413. I. Guyon, A. Elisseeff, An Introduction to Variable and Feature Selection, Journal of Machine Learning Research 3 (2003), 1157-1182. P.A. Flach, N. Lachiche, Confirmation-Guided Discovery of First-Order Rules with Tertius, Machine Learning 42 (2001), 61-95. Ian H. Witten and Eibe Frank "Data Mining: Practical machine learning tools and techniques",2nd Edition, Morgan Kaufmann, San Francisco, 2005.

Contact and Acknowledgement Intelligent Learning Object Guide 32 iLOG project website: http://cse.unl.edu/agents/ilog http://cse.unl.edu/agents/ilog Authors: S.A. Riley a, L.D. Miller a, L.-K. Soh a, A. Samal a, and G. Nugent b Email: [email protected], [email protected], [email protected], [email protected], [email protected] This material is based upon work supported by the National Science Foundation under Grant No. 0632642 and an NSF GAANN fellowship.

a framework for automatic empirically-based metadata generation intelligent learning object guide...

Documents