research and deployment of analytics in learning settings
DESCRIPTION
TRANSCRIPT
![Page 1: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/1.jpg)
Research and Deployment of Analytics in Learning Settings
Katrien Verbert���
PAWS Meeting 9 April 2012 School of Information Sciences, University of Pittsburgh
![Page 2: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/2.jpg)
Human-Computer Interaction
prof. Erik Duval
“Flexible Interaction between people and information”
Awareness & Sense-making
Computer Graphics
Language Intelligence & Information Retrieval
prof. Phil Dutré
prof. Sien Moens
http://hci.cs.kuleuven.be/
![Page 3: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/3.jpg)
![Page 4: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/4.jpg)
more focus on interaction...
![Page 5: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/5.jpg)
tracking traces
Rescuetime Rabbit- eclipse plugin
Blogs
![Page 6: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/6.jpg)
tracking traces
Rescuetime Rabbit- eclipse plugin
Blogs
![Page 7: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/7.jpg)
tracking traces
www.role-project.eu
![Page 8: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/8.jpg)
Duval, Erik. Attention please! Learning analytics for visualization and recommendation, Proceedings of LAK11: 1st International Conference on Learning Analytics and Knowledge, pages 9-17, ACM (2011)
![Page 9: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/9.jpg)
objectives
• self-monitoring for learners
• awareness for teachers
• learning resource use and recommendations
• part of Learning Analytics research [ACM LAK conf., Siemens 2011,
Duval 2011]
![Page 10: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/10.jpg)
overview
• Student Activity Meter
• Step Up!
• Recommender systems for learning
• Future research plans
![Page 11: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/11.jpg)
![Page 12: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/12.jpg)
Student activity meter (SAM): demo.
http://ariadne.cs.kuleuven.be/monitorwidget-rwtheval/ or http://bit.ly/I8AYV1
![Page 13: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/13.jpg)
Design Based Research Methodology
• Rapid prototyping
• Evaluate Ideas in short iteration cycles of Design, Implementation
& Evaluation
• Focus on Usefulness & Usability
• Think-aloud evaluations, SUS (System Usability Scale) surveys,
usability lab, ...
![Page 14: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/14.jpg)
Iteration one • usability and user satisfaction evaluation
• 12 CS students, using a -based
time tracker
• 2 evaluation sessions:
• task based interview with think aloud (after
1 week of tracking)
• user satisfaction (SUS & MSDT) (after 1
month)
![Page 15: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/15.jpg)
User satisfaction
• average SUS score: 73%
![Page 16: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/16.jpg)
iteration two
• 20 persons: 3 CGIAR, 2 Law, 8 CS teachers & 7
CS TA’s.
• An online survey about usefulness, teacher issues and how the tool can resolve these.
• on average: 40 mins are spent using SAM.
![Page 17: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/17.jpg)
CGIAR CASE STUDY
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗
?!✔
✔
✔ ?!
✔
?!?!
✔
✔ ✔ ?!?!✔
?!
✔
![Page 18: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/18.jpg)
demographics
evaluation goal
design changes
negative positive
I. 12 CS
students
usability, satisfaction, preliminary usefulness
1st iteration small usability
issues
• ↑learnability • ↓errors • good satisfaction • usefulness positive
II. 19
teachers & TA’s
assessing teacher needs,
use & usefulness
help function resource
recomm. not useful
• provides awareness • all vis. useful • many uses • 90% wants it
![Page 19: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/19.jpg)
iteration three
• open course on learning and knowledge
analytics, http://bit.ly/dWYVbX
• 12 visual analytics enthousiasts + experts (who
also teach)
• almost identical survey to CGIAR case.
![Page 20: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/20.jpg)
LAK CASE STUDY
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗ ✔
✔
✔
?!
?!
?!
✔
?!
✔ ?!
✔ ?!
✗
✗ ?!
?!
![Page 21: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/21.jpg)
ideas from experts
1
2
3
4
5 detailed information per student
the used resource types
detailed information of 2 students
detailed usage stats of resources
stats or vis. on content creation
![Page 22: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/22.jpg)
demographics
evaluation goal
design changes
negative positive
I. 12 CS
students
usability, satisfaction, preliminary usefulness
1st iteration small usability
issues
• ↑learnability • ↓errors • good satisfaction • usefulness positive
II. 19
teachers & TA’s
assessing teacher
needs, use & usefulness
help function resource
recomm. not useful
• provides awareness • all vis. useful • many uses • 90% want it
III. 12
participants
assessing teacher
needs, expert feedback, use & usefulness
re-orderable parallel
coordinates with
histograms
most addressed needs are indecisive
• provides awareness and feedback • many uses • 66% want it • recomm. can be useful
![Page 23: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/23.jpg)
Iteration four
• a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course
planner
• richer data set: tracking from programming
environment
• qualitative study using a structured face-2-face
interview ���
with 25 open questions.
![Page 24: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/24.jpg)
USER SATISFACTION
• average SUS score: 69,69%
all: want to continue using it 9/11: give it to students
![Page 25: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/25.jpg)
demo-graphics
evaluation goal
design changes
negative positive
I. 12 CS
students
usability, satisfaction, preliminary usefulness
1st iteration small usability
issues
• ↑learnability • ↓errors • good satisfaction • usefulness positive
II. 19
teachers & TA’s
assessing teacher needs, use & usefulness
help function resource
recomm. not useful
• provides awareness • all vis. useful • many uses • 90% want it
III. 12
participants
assessing teacher needs, expert
feedback, use & usefulness
re-orderable PC with
histograms
most addressed needs are indecisive
• provides awareness and feedback • many uses • 66% want it • recomm. can be useful
IV. 11
teachers & TA’s
use, usefulness & satisfaction
filter & search, icons, zooming in line chart,
editing PC axes
conflicting visions of
students doing well or at risk
• provides time overview • provides course overview • PC assist with detecting problems • many uses & insights • 100% want it
![Page 26: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/26.jpg)
conclusion
• SAM enables to find a wide variety of ���
new insights
• a better course overview
• understanding student time spending
• almost all participants want to continue
using SAM
26
![Page 27: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/27.jpg)
Santos Odriozola, Jose Luis; Govaerts, Sten; Verbert, Katrien; Duval, Erik Goal-oriented visualizations of activity tracking: a case study with engineering students, Proceedings of LAK12: 2nd
International Conference on Learning Analytics and Knowledge, pages 10, ACM (to appear)
![Page 28: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/28.jpg)
![Page 29: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/29.jpg)
Human-Computer Interaction Course
![Page 30: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/30.jpg)
http://bit.ly/I7hfbe
![Page 31: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/31.jpg)
usage
![Page 32: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/32.jpg)
User satisfaction
• average SUS score: 77%
![Page 33: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/33.jpg)
Nikos Manouselis, Hendrik Drachsler, Katrien Verbert and Erik Duval. Recommender Systems for Learning. SpringerBriefs in Computer Science, 90 pages, Springer US (to appear).
![Page 34: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/34.jpg)
![Page 35: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/35.jpg)
![Page 36: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/36.jpg)
http://bit.ly/A4CwZU
![Page 37: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/37.jpg)
challenges
• Evaluation
• Data sets
• Context
• User interfaces
![Page 38: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/38.jpg)
EVALUATION & DATA SETS
![Page 39: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/39.jpg)
Verbert, Katrien; Drachsler, Hendrik; Manouselis, Nikos; Wolpers, Martin; Vuorikari, Riina; Duval, Erik. Dataset-driven research for improving TEL recommender systems, LAK11:1st International
Conference on Learning Analytics and Knowledge, pages 44-53 (2011)
![Page 40: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/40.jpg)
http://bit.ly/acBKsp
![Page 41: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/41.jpg)
how to achieve objectives
• Setting up a website / maintain TELeurope group community
• Set up a open data repository for sharing educational datasets and
related researches outcomes
• Organizing annual workshop and SI
• Organizing a data competition like in TREC
![Page 42: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/42.jpg)
![Page 43: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/43.jpg)
dataTEL challenge & dataTEL cafe event
• a call for TEL datasets
• eight data sets submitted
http://bit.ly/ieqmWW
![Page 44: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/44.jpg)
http://dev.mendeley.com/datachallenge/
![Page 45: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/45.jpg)
Mendeley APOSDLE ReMashed Organic.edunet
Mace Melt
Collection period 1 year 3 months 2 years 9 months 3 years 6 months
Users 200.000 6 140 1.000 1.148 98
Items 1.857.912 163 96.000 11.000 12.000 1.923
Activities 4.848.725 1.500 23.264 920 461.982 16.353
reads + + - - + -
tags - (+) + + + +
ratings (+) - + + + +
downloads + + - - + +
search - + - - + -
collaborations - + - - - -
tasks/goals - + + - - -
sequence - + - - - -
competence - + - - + -
time - - - - + +
![Page 46: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/46.jpg)
Sam
Ian
Neil
A
B
C
high correlation
User-based CF
![Page 47: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/47.jpg)
Sam
Ian
Neil
A
B
C
high correlation
Item-based CF
![Page 48: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/48.jpg)
similarity measures
• Cosine similarity
• Pearson correlation
• Tanimoto or extended Jaccard coefficient
![Page 49: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/49.jpg)
similarity measures
MAE of item-based collaborative filtering based on different similarity metrics
![Page 50: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/50.jpg)
algorithms
MAE of user-based, item-based and slope-one collaborative filtering
![Page 51: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/51.jpg)
CONTEXT
![Page 52: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/52.jpg)
Verbert, Katrien; Manouselis, Nikos; Ochoa, Xavier ; Wolpers, Martin; Drachsler, Hendrik; Bosnic, Ivana; Duval, Erik. Context-aware recommender systems for learning: a survey and future challenges, IEEE
Transactions on Learning Technologies, 20 pages (Accepted)
![Page 53: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/53.jpg)
data dimensions
![Page 54: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/54.jpg)
![Page 55: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/55.jpg)
challenges
• context acquisition
• standardized representation of contextual data
• evaluation
• user interfaces
![Page 56: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/56.jpg)
VISUALIZING THE RATIONALE OF RECOMMENDATIONS���
![Page 57: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/57.jpg)
Visualizing recommendations
adapted from Keim et al. 2008
![Page 58: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/58.jpg)
objectives
• Address cold start issues
• Justification and trust
• Richer interaction capabilities
![Page 59: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/59.jpg)
examples
Klerkx and Duval 2009
O'Donovan et al. 2010
![Page 60: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/60.jpg)
Suggestions welcome!
![Page 62: Research and Deployment of Analytics in Learning Settings](https://reader038.vdocuments.us/reader038/viewer/2022110115/54bc2dd54a7959346d8b456b/html5/thumbnails/62.jpg)
References
• Duval, E. (2011). Attention please!: learning analytics for visualization and recommendation. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge, (pp. 9-17), ACM.
• D. Keim, G. Andrienko, J.-D. Fekete, C. Go ̈rg, J. Kohlhammer, and G. Melanc ̧on. Visual Analytics: Definition, Process, and Challenges. In A. Kerren, J. Stasko, J.-D. Fekete, and C. North, editors, Information Visualization, volume 4950 of Lecture Notes in Computer Science, pages 154–175. Springer Berlin / Heidelberg, 2008
• J. Klerkx and E. Duval. Visualising social bookmarks. Journal of Digital Information, 10(2):1–40, 2009
• J. O'Donovan, B. Gretarsson, S.Bostandjiev, C. Hall, and T. Hollerer. SmallWorlds: Visualizing Social Recommendations. In G. Melançon, T. Munzner, and D. Weiskopf (eds) Eurographics/ IEEE-VGTC Symposium on Visualization 2010, Volume 29 (2010), Number 3, 10 pages
• Siemens, G. & Gasevic, D. (eds) (2011). Proceedings of the 1st conference on Learning Analytics and Knowledge 2011. ACM.