Educational data mining overview & Introduction to Exploratory Data Analysis with DataShop

Ken Koedinger CMU Director of PSLC

Professor of Human-Computer Interaction & Psychology

Carnegie Mellon University

Overview

DataShop Overview Logging model DataShop Features

Quantitative models of learning curves Power law, logistic regression Contrasting KC models

Exploratory Data Analysis Exercise (start) Knowledge Component Model Editing

Logging & Storage Models

Education technologies are “instrumented” to produce log data

We encourage a standard log format XML format generalized from Ritter & Koedinger

(1995) Also convert log data from other formats

Relational Database -- complex!Relational Database -- complex!

Example activity generating “click stream” data

Geometry Cognitive Tutor: “Making Cans” problem Find the area of scrap metal left over after removing a circular area (the end of a can) from a metal

square. Student enters values in worksheet

Tutor provides feedback & instruction Records student’s actions & tutor responses

Logs stored in files on school server or database at Carnegie Learning Later imported into DataShop

DataShop logging model

Main constructs: Context message: the student, problem, and

session with the tutor Tool message: represents an action in the tool

performed by a student or tutor Tutor message: represents a tutor’s response to a

student action

DataShop XML format: Context message<context_message context_message_id="C2badca9c5c:-7fe5" name="START_PROBLEM"> <dataset> <name>Geometry Hampton 2005-2006</name> <level type="Lesson"> <name>PACT-AREA</name> <level type="Section"> <name>PACT-AREA-6</name> <problem> <name>MAKING-CANS</name> </problem> </level> </level> </dataset></context_message>

Dataset name

Course unit

Course section

Problem

DataShop XML format: Tool & Tutor Messages

<tool_message context_message_id="C2badca9c5c:-7fe5"> <semantic_event transaction_id="T2a9c5c:-7fe7" name="ATTEMPT" /> <event_descriptor> <selection>(POG-AREA QUESTION2)</selection> <action>INPUT-CELL-VALUE</action> <input>200.96</input> </event_descriptor></tool_message><tutor_message context_message_id="C2badca9c5c:-7fe5"> <semantic_event transaction_id="T2a9c5c:-7fe7" name="RESULT" /> <event_descriptor> … [as above] … </event_descriptor> <action_evaluation>CORRECT</action_evaluation></tutor_message>

Example Stored Transactions Student interactions (or transactions) are stored in a relational

database, can be exported as table Example: Student S01 on Making-Cans problem

Transactions

Info for each transaction student(s), session, time, problem, problem step,

attempt number, student action tutor response, number of hints, knowledge

component code Logging of on-line tools (e.g., a virtual lab)

does not include tutor response

Step & Transaction Definitions

A problem-solving activity typically involves many tool & tutor messages.

“Steps” represent completion of possible subgoals or pieces of a problem solution

“Transactions” are attempts at a step or requests for instructional help

Example: data aggregated by student-step

Overview

DataShop Overview Logging model DataShop Features

Quantitative models of learning curves Power law, logistic regression Contrasting KC models

Exploratory Data Analysis Exercise (start) Knowledge Component Model Editing

DataShop Analysis Tools

Dataset Info Performance Profiler Learning Curve Error Report Export Sample Selector

• Meta data for given dataset

• PI’s get ‘edit’ privileges, others must request it

• Meta data for given dataset

• PI’s get ‘edit’ privileges, others must request it

15

Papers and Files storage

Papers and Files storage

Dataset MetricsDataset Metrics

Problem Breakdown table Problem Breakdown table

Dataset Info

Performance Profiler

Aggregate by• Step• Problem• KC• Dataset Level

Aggregate by• Step• Problem• KC• Dataset Level

View measures of• Error Rate• Assistance Score• Avg # Hints• Avg # Incorrect• Residual Error Rate

View measures of• Error Rate• Assistance Score• Avg # Hints• Avg # Incorrect• Residual Error Rate

Multipurpose tool to help identify areas that are too hard or easy

Multipurpose tool to help identify areas that are too hard or easy

View by KC or Student, Assistance Score or Error Rate

View by KC or Student, Assistance Score or Error Rate

Time is represented on the x-axis as ‘opportunity’, or the # of times a student (or students) had an opportunity to demonstrate a KC

Time is represented on the x-axis as ‘opportunity’, or the # of times a student (or students) had an opportunity to demonstrate a KC

Visualizes changes in student performance over time

Visualizes changes in student performance over time

Learning Curve

• Provides a breakdown of problem information (by step) for fine-grained analysis of problem-solving behavior

• Attempts are categorized by student

• Provides a breakdown of problem information (by step) for fine-grained analysis of problem-solving behavior

• Attempts are categorized by student

View by Problem or KCView by Problem or KC

Error Report

Sample Selector

Filter by • Condition• Dataset Level• Problem• School• Student• Tutor Transaction

Filter by • Condition• Dataset Level• Problem• School• Student• Tutor Transaction

Easily create a sample/filter to view a smaller subset of data

Easily create a sample/filter to view a smaller subset of data

Shared (only owner can edit) and private samples

Shared (only owner can edit) and private samples

Export• Two types of export available

• By Transaction• By Step

• Anonymous, tab-delimited file• Easy to import into Excel!

You can also export the Problem Breakdown table and LFA values!

You can also export the Problem Breakdown table and LFA values!

Help/Documentation

• Extensive documentation with examples• Contextual by tool/report• http://learnlab.web.cmu.edu/datashop/help

• Extensive documentation with examples• Contextual by tool/report• http://learnlab.web.cmu.edu/datashop/help

Glossary of common terms, tied in with PSLC Theory wiki

Glossary of common terms, tied in with PSLC Theory wiki

New Features

Manage Knowledge Component models Create, Modify & Delete KC models within

DataShop Addition of Latency Curves to Learning Curve

Reporting Time to Correct Assistance Time

Problem Rollup & Export Enhanced Contextual Help

Overview

DataShop Overview Logging model DataShop Features

Quantitative models of learning curves Power law, logistic regression Contrasting KC models

Exploratory Data Analysis Exercise (start) Knowledge Component Model Editing

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Recall learning curve story

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Without decomposition, using just a single “Geometry” KC,

no smooth learning curve.

But with decomposition, 12 KCs for area concepts,

a smooth learning curve.

Upshot: A decomposed KC model fits learning & transfer data better than a “faculty theory” of mind

Learning curve analysis

The Power Law of Learning (Newell & Rosenbloom, 1993) Y = a Xb

Y – error rateX – opportunities to

practice a skilla – error rate on 1st opportunity b – learning rateAfter the log transformation“a” is the “intercept” or starting point of the learning curve“b” is the “slope” or steepness of the learning curve

More sophisticated learning curve model Generalized Power Law to fit learning curves

Logistic regression (Draney, Wilson, Pirolli, 1995)

Assumptions Different students may initially know more or less

=> use an intercept parameter for each student Students learn at the same rate

=> no slope parameters for each student Some productions may be more known than others

=> use an intercept parameter for each production Some productions are easier to learn than others

=> use a slope parameter for each production

These assumptions are reflected in detailed math model …

More sophisticated learning curve model

Probability of getting a step correct (p) is proportional to:- if student i performed this step = Xi,

add overall “smarts” of that student = i

- if skill j is needed for this step = Yj, add easiness of that skill = j

add product of number of opportunities to learn = Tj & amount gained for each opportunity = j

( ) jjjjjiipp TYYX ∑ ∑∑ ++=− γβα1ln p

Use logistic regression because response is discrete (correct or not) Probability (p) is transformed by “log odds” “stretched out” with “s curve” to not bump up against 0 or 1

(Related to “Item Response Theory”, behind standardized tests …)

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Different representation, same model Predicts whether student is correct depending on knowledge & practice Additive Factor Model (Draney, et al. 1995, Cen, Koedinger, Junker, 2006)

The Q MatrixThe Q Matrix

How to represent relationship between knowledge components and student tasks?

Tasks also called items, questions, problems, or steps (in problems) Q-Matrix (Tatsuoka. 1983)

2* 8 is a single-KC item 2*8 – 3 is a conjunctive-KC item, involves two KCs

29

Item | KC Add Sub Mul Div

2*8 0 0 1 0

2*8 - 3 0 1 1 0

30

Model Evaluation

• How to compare cognitive models?• A good model minimizes prediction risk by balancing fit

with data & complexity (Wasserman 2005)• Compare BIC for the cognitive models

• BIC is “Bayesian Information Criteria”• BIC = -2*log-likelihood + numPar * log(numOb)• Better (lower) BIC == better predict data that haven’t seen

• Mimics cross validation, but is faster to compute

31

Model Title LL BIC numPar

G-2,175 4,566 26

Original -1,911 4,271 54

Item -1,720 5,554 254

• Data: the Geometry Area Unit• 24 students, 230 items, 15 KCs

Learning curve constrast in Physics dataset …

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Not a smooth learning curve -> this knowledge component model is wrong. Does not capture genuine student difficulties.

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

More detailed cognitive model yields smoother learning curve. Better tracks nature of student difficulties & transfer

(Few observations after 10 opportunities yields noisy data)

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Best BIC (parsimonious fit) for Default (original) KC model

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Better than simpler Single-KC model

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

And better than more complex Unique-step (IRT) model

Overview

DataShop Overview Logging model DataShop Features

Quantitative models of learning curves Power law, logistic regression Contrasting KC models

Exploratory Data Analysis Exercise (start) Knowledge Component Model Editing

Exploratory Data Analysis Exercise Goals:

1) Get familiar with data 2) Learn/practice Excel skills

Tasks: 1) create a “step table” 2) graph learning curves

TWO_CIRCLES_IN_SQUARE problem: Initial screen

TWO_CIRCLES_IN_SQUARE problem: An error a few steps later

TWO_CIRCLES_IN_SQUARE problem: Student follows hint & completes prob

Exported File Loaded into Excel

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

See handout of exercise …Do some of in next session

Overview

DataShop Overview Logging model DataShop Features

Quantitative models of learning curves Power law, logistic regression Contrasting KC models

Exploratory Data Analysis Exercise (start) Knowledge Component Model Editing

DataShop Demo

Examples of exercise

KC model editing

END