evaluating the student activity meter: two case studies
DESCRIPTION
presented at ICWL 2011, Hong Kong, 8-9 December 2011.TRANSCRIPT
EVALUATING THE STUDENT ACTIVITY METER:
TWO CASE STUDIES.Sten Govaerts, Katrien Verbert & Erik Duval
http://www.role-project.eu
1
Thursday 8 December 11
OVERVIEW
• Problem statement
•Objectives
•Design & Implementation
• Evaluations
• Conclusion & Future work
2
Thursday 8 December 11
PROBLEM
3
Thursday 8 December 11
PROBLEM
3
Thursday 8 December 11
OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
• learning resource use and recommendations
4
Thursday 8 December 11
STUDENT ACTIVITY METER (SAM): DEMO.
5
Thursday 8 December 11
THREE EVALUATIONS
• with CS students• ICWL 2010 paper: “Visualizing Activities for
Self-reflection and Awareness”
• with CGIAR courses and teachers
• with Learning and Knowledge Analytics course participants
6
Thursday 8 December 11
2010 PAPER
• usability and user satisfaction evaluation with 12 CS students
• in general, people understand the visualizations well!
• solved some issues and requests since
Thursday 8 December 11
2010 PAPER
• usability and user satisfaction evaluation with 12 CS students
• in general, people understand the visualizations well!
• solved some issues and requests since
Thursday 8 December 11
2010 PAPER
• usability and user satisfaction evaluation with 12 CS students
• in general, people understand the visualizations well!
• solved some issues and requests since
Thursday 8 December 11
2010 PAPER
• usability and user satisfaction evaluation with 12 CS students
• in general, people understand the visualizations well!
• solved some issues and requests since
Thursday 8 December 11
METHODOLOGY.
• design-based research: give students & teachers a tool and evaluate
• usability studies
• paper mockups
• surveys
8
Thursday 8 December 11
CASE STUDY
• 20 persons: 3 CGIAR, 2 Law, 8 CS teachers & 7 CS TA’s.
• An online survey about usefulness, teacher issues and how the tool can resolve these.
• on average: 40 mins are spent using SAM.
9
Thursday 8 December 11
CGIAR CASE STUDY
10
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗
?✔
✔
✔?
✔
??
✔
✔✔??✔?
✔
Thursday 8 December 11
CGIAR CASE STUDY
10
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗
?✔
✔
✔?
✔
??
✔
✔✔??✔?
✔
Thursday 8 December 11
CGIAR CASE STUDY
11
Bar chartParallel coordinatesLine chart
✔✔✔
Thursday 8 December 11
CGIAR CASE STUDY
12
Thursday 8 December 11
CGIAR CASE STUDY
good indicator for effortunderstand the workload
use for course design optimization
obtain course overview
increase awareness
detect outliers
progress evolutioncompare students
want better 1 to 1 comparison tool
more details on student
want to search for students
more metricswanted
12
Thursday 8 December 11
CGIAR CASE STUDY
good indicator for effortunderstand the workload
use for course design optimization
obtain course overview
increase awareness
detect outliers
progress evolutioncompare students
want better 1 to 1 comparison tool
more details on student
want to search for students
more metricswanted
12
Thursday 8 December 11
CASE STUDY
• open course on learning and knowledge analytics, http://bit.ly/dWYVbX
• 12 visual analytics enthousiasts + experts (who also teach)
• almost identical survey to CGIAR case.
13
Thursday 8 December 11
CASE STUDY
• open course on learning and knowledge analytics, http://bit.ly/dWYVbX
• 12 visual analytics enthousiasts + experts (who also teach)
• almost identical survey to CGIAR case.
13
Thursday 8 December 11
LAK CASE STUDY
14
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗✔
✔
✔?
?
?
✔?
✔?
✔?
✗
✗?
?
Thursday 8 December 11
LAK CASE STUDY
14
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗✔
✔
✔?
?
?
✔?
✔?
✔?
✗
✗?
?
Thursday 8 December 11
LAK CASE STUDY
15
Bar chartParallel coordinatesLine chart
???
Thursday 8 December 11
LAK CASE STUDY
16
Thursday 8 December 11
LAK CASE STUDY
chronological course dwell time
verify the status of the classroom activity
self-reflection to measure progress and increase
motivation find students experiencing problems and low
engagement
more metrics
more data16
Thursday 8 December 11
LAK CASE STUDY
chronological course dwell time
verify the status of the classroom activity
self-reflection to measure progress and increase
motivation find students experiencing problems and low
engagement
more metrics
more data16
Thursday 8 December 11
IDEA’S FROM EXPERTS
17
detailed information per student
the used resource types
detailed information of 2 students
detailed usage stats of resources
stats or vis. on content creation
Thursday 8 December 11
IDEA’S FROM EXPERTS
1
2
3
4
5
17
detailed information per student
the used resource types
detailed information of 2 students
detailed usage stats of resources
stats or vis. on content creation
Thursday 8 December 11
CONTINUED WORK...
18
Thursday 8 December 11
CONTINUED WORK...
18
Thursday 8 December 11
FUTURE WORK
• extend SAM with feedback of experts
• evaluate the usefulness in a larger group of students
• evaluate with teachers during and after a course.
19
Thursday 8 December 11
YOUR DATA IN SAM ?!
20
Thursday 8 December 11
2
slides will appear on http://www.slideshare.net/stengovaerts
slides will appear on http://www.slideshare.net/stengovaerts 21
Thursday 8 December 11