mhci pslc data shop project

MHCI PSLC Data Shop Project

Final Design Presentation

MHCI PSLC Team | Summer 2005 2

The Team

Jason HumSandi Lowe

Meghan Myers

Sam Zaiss

Jeff Wong


Outline

I. Background: PSLC & The Data Shop

II. The MHCI ProjectIII. Use Case ScenarioIV. Prototype DemoV. Implementation TimelineVI. Wrap Up


Outline




What is the PSLC?

Collect Process Access

Pre-Defined

Free-Form

PSLCLearnLabData Shop

Export


PSLC Goals

• Further current education research• Enable new education research• Support collaboration• Support 7 LearnLab courses


What is the Data Shop?

Collect Process Access

Pre-Defined

Free-Form

PSLCLearnLabData Shop

Export


Outline




The MHCI Data Shop Project

Access

Pre-Defined Free-Form

Export

LearningCurves

ErrorReport

ProblemProfile

DataExport

SessionBrowser

TimelineViz

BehaviorGraphHelp FX


Project Requirements

• High-Fidelity Proof-of-Concept Prototype• In-Depth Research Including Weekly

User Testing– 13 Contextual Inquiries– 8 Requirements Interviews– 12 Competitive Analyses– 37 User Tests

• Deliverables– Current Prototype– Requirements Document– Design Specification– Supporting Data– Design Iterations


MHCI Project Timeline

• Began With Low Fidelity Paper Prototypes• Gradually Added Features & Increased

Fidelity• Weekly User Testing Throughout

Iteration

UserTesting

Summer WorkshopStart of Summer

Paper PrototypesMid-Fi Prototypes

Hi-Fi Prototype

End


Project Themes

• Context Matters• Facilitate Inter-Report Navigation• Create Specialized Reports• Emphasize Visual Communication


Outline




ITS Background

• Intelligent Tutoring Systems (ITSs) used:– To help students learn– To gain insight into how students learn

• Consist of a series of problems in a particular subject.– Order of problems is random– Each problem composed of a number of

steps; each of which test one or more knowledge components


Meet Dave Jargenson

• Interested in how students learn Algebra.• Has been doing education research for 10 years.• The PSLC’s newest member, he has already run a

study with the Center.• Trying out the Data Shop with a couple basic studies.

Name: David Jargenson, Ph.D.Age: 35Affiliation: Research Scientist, Wisconsin

Center for Education Research


Dave’s Data Exploration

Determine Most Frequent Unexpected Error

Determine How Students Are Performing Overall

Compare Performance of 2 Samples

Select Certain Transactions from the Data ShopFor Further Analysis


Outline




Error Report• Supporting Data:

– Requirements Solicitation (VanLehn, Koedinger, Ritter)– General Research Contextual Inquiries (U3, U8, U10)– LearnLab Research Contextual Inquiries (U11, U12)

• Goal:– View all mistakes that students made, by

frequency, steps and knowledge components


Error Report

"Here we are. Errors by classification. Hmm,

unanticipated? Oh, I can click it. Okay, the most common one is

miles.“-

U30


Learning Curves• Supporting Data:

– Requirements Solicitation (Koedinger, Aleven, Ritter)

– General Research Contextual Inquiries (U2)– LearnLab Research Contextual Inquiries (U13)

• Goal: understand students’ performance over time particular knowledge components


Learning Curves/Problem Profiles• “Oh! I don’t need to see the [transaction

table], it’s right here in the graph.” (-U28)

• “I like being able to see the curves without punching in the formulas.” (-U1)

• “I love that the a’s and the b’s come right up.” (-U36)

• “I liked the Problem Profile. Leave it as it is.”(-U28)


Sample Selector

• Supporting Data: – Contextual Inquiry Research: U3, U15– Requirements Solicitation: Aleven, Koedinger– User Testing Pre-test Questions: U15, U16,

U19-U24, U26, U28-U34, U36

• Goal: Define multiple groups of students and compare their performance throughout the standard reports within the Data Shop


Sample Selector

• "so the only option I have is 'all students.' Ah, but I can edit this list.“ (U27)

• “Oh, maybe I need to create a new sample.” (U31)

• “making the samples was fairly easy.” (U29)


Data Export

• Dave’s Goal is to:– Select a subset of his data– Export it to a file for further analysis

• Supporting Data:– Requirements Solicitation (Koedinger, VanLehn)– General Research Contextual Inquiries (U5, U8)


Data Export Questions


Outline




Implementation planEstimated Implementation Time (in

Days)


Outline




Context Matters

• Intimate Knowledge of Tutors Required

• Dig a little deeper, right away


Inter-report Navigation

• Reports are useful when they are connected.

• Carry context between reports when possible.


Specialized Reports

• More tailored than a stat package• Eliminates grunt work


Use Visual Communication

• Get familiar with data quickly• Identify points of interest


Evolution of Error Report



"I really don't understand what it [error report] means. When Ken was showing it to us [earlier in the day at the Data Shop demo] it made sense but on my own I wasn't sure." –User 13, Summer School



"This wasn't very helpful... probably just the layout - it's hard to decipher….this is very difficult to read… I’m not sure what these errors mean." – User 13


Acknowledgements

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture. QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Kurt Van LehnVincent Aleven

Ken Koedinger

Polo ChauShipra Kayan

Braden KowitzPeter Centgraf

Bonnie John

Carolyn Rose

Bob Kraut

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Michael Bett

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Ben Billings

Andrea Knight

Alida Skogsholm


The End

“It’s like having my very own grad student!” – User 21


Backup Slide: Data Visualization• Information Visualization (Card, 2003)

says that Data Visualization improves cognition in 6 ways:– Increasing the memory and processing

resources available to users– Reducing the search for information– Using visual representations to enhance the

detection of patterns– Enabling perceptual inference operations– Using perceptual attention mechanisms for

monitoring– Encoding information in a manipulable medium


Backup Slide - Error Report• Horizontal Stacked

Bars– Option to take hints as

errors of omission– Allows them to compare

“down the line”– Error names fit better

horizontally– Visualization provides

better performance than tables


Problem Profile

• Supporting Data:– General Research Contextual Inquiries (U1, U3, U7, U8,

U9, U10)– LearnLab Research Contextual Inquiries (U11, U12)– Course Committee Survey (Chem)– Think Aloud Pilot (U2)

• Goal: Understand students’ performance on a particular problem, and the problem’s context


Multi-Selection(in long lists)

• A more standard method of indicating multiple-selection

• Highlight helps users quickly spot which items are selected if scrolling the list.

“…if there were check boxes on the side I would have known I could select more than one” – (U15)


“Scrubbing” A method to quickly

compare across knowledge components

“Oh interesting. It [the next curve] pops up.“

– (U35) “ooh…that's so cute….I'm

going to click on that point to see why it jumped back up.”

– (U36)


Why Not Just Add Condition?

• Within-subject experiments require the capability for students to be assigned to multiple conditions

• Sample Selector allows for multiple groupings based on individual researchers’ units of analysis


What About Behavior Characteristics?

• The Sample Selector can build groups of students based on any characteristic in the database

• Student behavior characteristics are not currently fields in the database

• Once explicitly defined and included in the database, any behavior characteristic can be added and then used to build samples


Student Characteristics


Problem Characteristics


Step Characteristics


Data Export: Needs Served

• Escape hatch– Users can do whatever they want

• Narrow down data– Export only rows which are relevant– Export only columns which are

relevant.– User tables tend to be very wide.– Users tend to copy out only the

columns they need and move them to different worksheets.


Why choose columns?• Columns are defined by Data Shop

architecture– All may not be relevant to every study

• You fit only the columns you are interested in within the width of the table. – No side scrolling or rearranging of

columns


Direct Manipulation• Some direct manipulation of the

table in Data Export

• Users 5, 11, and 27 tried to manipulate the graph directly.

• Having the pop-up menu instantly tells the user what clicking on the graph means.


Simple Filtering

• We allow simple filtering on each column

• Easy-to-use complex querying is an open problem

• Complex querying is better done with existing query languages such as SQL.


Main Screen


Filter Dialog


Export Dialog

mhci pslc data shop project

Documents

error reportsupporting

mhci project timelinebegan

research scientist

students learnconsist

students learnto gain

wisconsin center

order of problems

series of problems