mhci pslc data shop project spring progress report
Post on 20-Dec-2015
216 views
TRANSCRIPT
MHCI PSLC Data Shop Project
Spring Progress Report
MHCI PSLC Team | Spring 2005 2
The Team
Jason HumDesign Lead
Sandi LoweProject Manager
Meghan MyersClient Liaison
Sam ZaissDocuments Manager
Jeff WongTech Lead
MHCI PSLC Team | Spring 2005 3
Outline
I. What is the PSLC?II. What is the Data Shop?III. Analyzing DataIV. Exporting DataV. Project ThemesVI. Preliminary DesignVII. Project PlanVIII.Final Takeaways
MHCI PSLC Team | Spring 2005 4
What is the PSLC?
Collect Process Access
Pre-Defined
Free-Form
PSLCLearnLab
Data Shop
Export
MHCI PSLC Team | Spring 2005 5
Background on PSLC
• Further current education research• Enable new education research• Support collaboration• Support LearnLab courses
MHCI PSLC Team | Spring 2005 6
Background on LearnLab Courses
• 7 course areas under PSLC– 3 Language (Chinese, French, ESL)– 2 Math (Algebra, Geometry)– 2 Science (Physics, Chemistry)
• Various stages of development
MHCI PSLC Team | Spring 2005 7
What is the Data Shop?
Collect Process Access
Pre-Defined
Free-Form
LearnLab
Data Shop
Export
PSLC
MHCI PSLC Team | Spring 2005 8
Collecting the Data
• Allows researchers to set up an in-vivo LearnLab experiment
• Facilitates logging of data in these experiments
MHCI PSLC Team | Spring 2005 9
Processing the Data
• Strip away student information• Perform computations on data to
streamline analysis• Create supporting data tables
MHCI PSLC Team | Spring 2005 10
Accessing the Data
• Built as a web application• Selected data analysis techniques• Data filtering and exportation
MHCI PSLC Team | Spring 2005 11
Analyzing Data: Pre-Defined Reports
Collect Process Access
Pre-Defined
Free-Form
LearnLab
Data Shop
Export
PSLC
MHCI PSLC Team | Spring 2005 12
Learning Curves
• Displays students’ performance over time on a particular knowledge component
• Log Files must include:– Knowledge Components– Student performance on individual
opportunities– Match Knowledge Components to
Problems / Sub-Goals
• Supporting Data:– Requirements Solicitation
(Koedinger, Aleven, Ritter)– General Research Contextual
Inquiries (U2)– LearnLab Research Contextual
Inquiries (U13)
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Courtesy of Data Shop
MHCI PSLC Team | Spring 2005 13
Help Effectiveness Report
• The probability of the Section getting task or sub-goal x incorrect after a particular help message
• Log Files must include:– Help / Hints– Whether a student received help,
and if the answer that followed was correct or incorrect
– Measure of student’s knowledge before help message
• Supporting Data:– Requirements Solicitation
(VanLehn, Ritter)– General Research Contextual
Inquiries (U1, U10)– LearnLab Research Contextual
Inquiries (U11, U12)
MHCI PSLC Team | Spring 2005 14
Error Report
• Determine common mistakes that students make on tasks • Log Files must include:
– Erroneous Input & Tutor Response to Error– Error Classifications
• Supporting Data:– Requirements Solicitation (VanLehn, Koedinger, Ritter)– General Research Contextual Inquiries (U3, U8, U10)– LearnLab Research Contextual Inquiries (U11, U12)
MHCI PSLC Team | Spring 2005 15
Problem Profile
• Context and student performance on a particular problem • Log Files must include:
– Problem Text– Record of students’ answers on the problem and whether the answers
were correct or incorrect
• Supporting Data:– General Research Contextual Inquiries (U1, U3, U7, U8, U9, U10)– LearnLab Research Contextual Inquiries (U11, U12)– Course Committee Survey (Chem)– Think Aloud Pilot (U2)
MHCI PSLC Team | Spring 2005 16
Analyzing Data: Free-Form Analysis
Collect Process Access
Pre-Defined
Free-Form
LearnLab
Data Shop
Export
PSLC
MHCI PSLC Team | Spring 2005 17
Timeline Visualization
• Visualizes the pattern of each student’s actions
• Log Files must include:– Time measurements (ms)
• Supporting Data:– Users want to drill down and tell
stories– General Research Contextual
Inquiries (U1, U5, U8, U9, U10)
Courtesy of Andrea Knight
MHCI PSLC Team | Spring 2005 18
Session Browser
• Visualizes the pattern and context of each student’s actions • Log Files must include:
– Time measurements (ms)– Event Hierarchy
• Supporting Data:– Users want to drill down and tell stories– Requirements Solicitation (VanLehn)– General Research Contextual Inquiries (U1, U3, U4, U5, U7, U8, U9, U10)
Courtesy of Jack Mostow
MHCI PSLC Team | Spring 2005 19
Event Filtering (aka Rollup Table)
• Allows researcher to specify their variables and calculations over those variables
• Supporting Data:– Requirements Solicitation (Koedinger, VanLehn)– General Research Contextual Inquiries (U5, U8)– U5: “By the time I get the data I need out of the system, I’ve
forgotten the question I wanted to ask”
MHCI PSLC Team | Spring 2005 20
Exporting Data
Collect Process Access
Pre-Defined
Free-Form
LearnLab
Data Shop
Export
PSLC
MHCI PSLC Team | Spring 2005 21
Project Themes
Collect Process Access
Pre-Defined
Free-Form
LearnLab
Data Shop
Export
PSLC
MHCI PSLC Team | Spring 2005 22
Project Themes
1. Support All LearnLab Tutors
Problem SubgoalKnowledge Component Right Answers Wrong Answers Unanticipated Errors Wrong Answer Feedback Hint Error Clusters
Algebra ? ? ? ? ? ? ? ? ?Chemistry
Chinese ? ? ? ? ? ? ? ? ?ESL ? ? ? ? ? ? ? ? ?
French ? ? ? ? ? ? ? ? ?Geometry –
Physics / -
Concept in Tutor
– Concept not in Tutor
? Not enough information
MHCI PSLC Team | Spring 2005 23
Project Themes
2. Facilitate Inter-Report Navigation
MHCI PSLC Team | Spring 2005 24
Project Themes
3. Let Researchers See Only the Data They Want to See, When They Want to See It.– Reports show only what’s relevant– Find out how researchers want to narrow
down their data in reports– Minimize interface elements for each
report
MHCI PSLC Team | Spring 2005 25
Preliminary Design
MHCI PSLC Team | Spring 2005 26
Design for Theme 2
• Large buttons to move between reports • Key Considerations:
– Data transfer from report to report– Saving reports with interesting data along the way– Issues of scale
MHCI PSLC Team | Spring 2005 27
Design for Theme 3
• Specialized filtering for each report • Key Considerations:
– Condensing report specification & modification into one set of controls
– Distinguishing between singular & multiple selection
– Enabling fast selection
• Design Support:– List boxes for long lists of variables (StatTransfer)– Clear representation of curriculum hierarchy
(Froogle)
• Design Challenges:– Minimizing space while maximizing speed– Providing sufficient context to the user
MHCI PSLC Team | Spring 2005 28
User Testing Findings
• 2 users doing an analysis with learning curves– 1st year HCI researcher PhD, familiarity with tutor
studies– 4th year education researcher PhD, no tutor studies
• Needs discovered:– Compare curves against pre- and post-tests– Show p-value and regression equation– Describe sample– Click on a Learning Curve spike to understand
context– Show text of the problem
MHCI PSLC Team | Spring 2005 29
Project Plan
Pre-Summer Workshop• Add breadth to current structure
– Error Reports– Help Effectiveness– Problem Profiles
• Develop Filtering Designs (Mid-Fi)• Prototype Session Browser & Timeline
Visualization (Low-Fi)• Heuristic Evaluation• User Testing
Pre-Workshop: 6 wks Post-Workshop: 5 wks1 wk
MHCI PSLC Team | Spring 2005 30
Project Plan
Summer Workshop (Week of June 27th)• High-Intensity User Testing, Iteration
Post-Summer Workshop• Database Connections for Pre-Defined Reports• Session Browser & Timeline Visualization (Mid-Fi)• User Testing• Revisions
Pre-Workshop: 6 wks Post-Workshop: 5 wks1 wk
MHCI PSLC Team | Spring 2005 31
Final Takeaways
1. Rules for Data Sharing– Sharing outside research group
• Competitive Analysis revealed need for sharing regulations
• U5: “I’m happy to share my data if the researchers follow PSLC’s guidelines”
• For now, no open sharing
– Sharing inside research group• Seen as a need in General Research CIs (U1 -
U8)• Designed & tested by end of summer
MHCI PSLC Team | Spring 2005 32
Final Takeaways
2. Context Matters– Understanding data is hard without context– Issues discovered to date:
• Complex naming => complex design (CA) • Tutor writers use intimate knowledge of the
tutor frequently (U5, U7, U9, U11, U12)• Researchers unfamiliar to the tutor need
that intimate knowledge (TA Pilot Test)
– TA Pilot Test U2: “So you’d have… a master book that would tell you what all the problems are? Because I’d never remember.”
MHCI PSLC Team | Spring 2005 33
Final Takeaways
3. Courses in Development Should Log Necessary Data
– Survey course committees as needed– Proposed reports will function with
current PSLC tutors– Can’t predict what future tutors may and
may not log
MHCI PSLC Team | Spring 2005 35
Behavior Graph
• Illustrates the strategies students are using to solve problems
• Log Files must include:– Sub-goals
• Supporting Data:– Requirements Solicitation (Koedinger)– General Research Contextual Inquiries
(U8, U10)– Course Committee Survey (Chem)– Think Aloud Pilot (U2)
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Courtesy of Data Shop
MHCI PSLC Team | Spring 2005 36
Physics Survey
Tutor Element ANDES WhyAtlas
Problem Problem Task
Sub goal Entries Expectations / Responses
Knowledge Comp Principles Principles
Right Answers Right Answers Correct Features
Wrong Answers Recognized Errors Recognized Errors
Unanticipated Errors Unrec. Errors Unrec. Errors
Wrong Ans Feedback
Red Incorrect Error State
Hint Next Step Help / What’s Wrong Help
N/A
Error Cluster Error Classes Misconceptions
MHCI PSLC Team | Spring 2005 37
Chinese/French Survey
Tutor Element Chinese/French
Problem Activities
Sub goal N/A
Knowledge Comp Skills
Right Answers Correct Answers
Wrong Answers Incorrect Answers
Unanticipated Errors Unpredicted Errors
Wrong Ans Feedback
Error Feedback
Hint Funtional / Contextual Help
Error Cluster (e.g., Orthography, Meaning, Form)