analytics - presentation in dkit
TRANSCRIPT
![Page 1: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/1.jpg)
Learning Analytics - value what we measure or measure what we value
Dr Mark Glynn
@glynnmark
![Page 3: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/3.jpg)
Outline
• Introduction• Motivation and goals• Challenges• Examples• Technical bits• Discussion
– What would you like to analyse
– Collaboration
![Page 4: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/4.jpg)
Teaching Enhancement Unit
TEACHINGENHANCEMENT
UNIT
Onlineand Blended Learning
Support
Awards and Grants
Credit Earning Modules
Professional Development
Workshops
![Page 5: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/5.jpg)
Data Analytics
Data analytics is the science of extracting actionable insight from large amounts of raw data
DIT – MSc in Computing
![Page 6: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/6.jpg)
Youtube
![Page 7: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/7.jpg)
Tesco
![Page 8: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/8.jpg)
Definiton
Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs. A related field is educational data mining.
- wikipedia
![Page 9: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/9.jpg)
Discussion
What data do we already collect?
![Page 10: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/10.jpg)
So much student data we could use
Demographics• Age, home/term address, commuting distance, socio-economic status, family composition, school
attended, census information, home property value, sibling activities, census information
Academic Performance• CAO and Leaving cert, University exams, course preferences, performance relative to peers in
school
Physical Behaviour• Library access, sports centre, clubs and societies, eduroam access yielding co-location with others
and peer groupings, lecture/lab attendance,
Online Behaviour• Mood and emotional analysis of Facebook, Twitter, Instagram activities, friends and their actual
social network, access to VLE (Moodle)
![Page 11: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/11.jpg)
Motivation
![Page 12: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/12.jpg)
Discussion
What challenges do you foresee in your institution?
![Page 13: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/13.jpg)
Core Principles – Open University UK
Learning analytics is a moral practise which should align with core organisational principles
The purpose and boundaries regarding the use of learning analytics should be well defined and visible
Students should be engaged as active agents in the implementation of learning analytics
The organisation should aim to be transparent regarding data collection and provide students with the opportunity to update their own data and consent agreements at regular
intervalsModelling and interventions based on analysis of data should be free from bias and
aligned with appropriate theoretical and pedagogical frameworks wherever possible
Students are not wholly defined by their visible data or our interpretation of that data
Adoption of learning analytics within the organisation requires broad acceptance of the values and benefits (organisational culture) and the development of appropriate skills
The organisation has a responsibility to all stakeholders to use and extract meaning from student data for the benefit of students where feasible
![Page 14: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/14.jpg)
Amazon
![Page 15: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/15.jpg)
Examples
![Page 16: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/16.jpg)
The earlier we diagnose, the earlier we can treat
John CarrollMark GlynnEabhnat Ni Fhloinn
@glynnmark
![Page 17: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/17.jpg)
Maths Diagnostic test
![Page 18: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/18.jpg)
Data Analytics on VLE Access DataHow much can we mine from a mouseclick ?
John BrennanOwen CorriganAly EganMark GlynnAlan F. SmeatonSinéad Smyth
@glynnmark
![Page 19: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/19.jpg)
No significant difference in the entry profiles of participants vs. non-participants overall
PredictEd Participant Profile
![Page 20: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/20.jpg)
Total Moodle Activity – notice the periodicity
![Page 21: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/21.jpg)
One example module – ideal !
![Page 22: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/22.jpg)
LGxxx – Predictor confidence (ROC AUC)
![Page 23: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/23.jpg)
All modules
![Page 24: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/24.jpg)
LG116 MS136 LG101 HR101 LG127 ES125 BE101 SS103 CA103 CA1680%
20%
40%
60%
80%
100%WorkshopsWikisForumsAssignmentsQuizzesscormlessonchoicefeedbackdatabaseglossarywikiurlbookpagesfoldersfiles
Course content
a b c d e f g h i j
![Page 25: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/25.jpg)
Study by numbers
• 17 Modules across the University (first year, high failure rate, use Loop, periodicity, stability of content, Lecturer on-board)
• Offered to students who opt-in or opt-out, over 18s only
• 76% of students opted-in, 377 opted-out, no difference among cohorts
• 10,245 emails sent to 1,184 students who opted-in over 13 weekly email alerts
![Page 26: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/26.jpg)
The Interventions – Lecturers’ Experience
![Page 27: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/27.jpg)
Student Engagement - individual
![Page 28: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/28.jpg)
Student Engagement - individual
![Page 29: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/29.jpg)
Regression analysis- class grades
![Page 30: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/30.jpg)
What to measure?
Content Transfer
Interaction Collabora-tion
Assessment
![Page 31: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/31.jpg)
Collaboration
![Page 32: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/32.jpg)
Modules which work well …
• Have periodicity (repeatability) in Moodle access• Confidence of predictor increases over time• Don't have high pass rates (< 0.95)• Have large number of students, early-stage
![Page 33: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/33.jpg)
LGxxx: law based subject
Students / year = ~110Pass rate = 0.78
![Page 34: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/34.jpg)
Student Interventions: Feedback
![Page 35: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/35.jpg)
Relative data
![Page 36: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/36.jpg)
Student Experience of PredictED
Students who took part were asked to complete a short survey at the start of Semester 2 - N=133 (11% response rate)
Question Group 1 (more detailed email)
Group 2
% of respondents who opted out of PredictED during the course of the
semester4.5% 4.5%
% who changed their Loop usage as a result of the weekly emails
43.3% 28.9%
% who would take part again/are offered and are taking part again
72.2% (45.6%/ 26.6% )
76.6% (46% /30.6% )
![Page 37: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/37.jpg)
33% said they changed how they used Loop. We asked them how?
• Studied more– “More study”– “Read some other articles online”– “Wrote more notes”– “I tried to apply myself much more, however yielded no results”– “It proved useful for getting tutorial work done”
• Used Loop more– “I tried harder to engage with my modules on loop”– “I think as it is recorded I did not hesitate to go on loop. And loop as become
my first support of study.”– “I logged on more”– “I read most of the extra files under each topic, I usually would just look at
the lecture notes.”– “I looked at more of the links on the course nes pages, which helped me to
further my understanding of the topics”– “I learnt how often I need to log on to stay caught up.”
![Page 38: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/38.jpg)
Did you change Loop usage for other modules?
• Most who commented used Loop more often for other modules– “More often”– “More efficient”– “Used loop more for other modules when i was logging onto
loop for the module linked to PredictED”– “Felt more motivated to increase my Loop usage in general
for all subjects”
One realised that Lecturers could see their Loop activity“I realised that since teachers knew how much i was
using loop, i had to try to mantain pages long on so it looked as if i used it a lot”
![Page 39: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/39.jpg)
Subject Description Non-Participant ParticipantBE101 Introduction to Cell Biology and Biochemistry 58.89 62.05CA103 Computer Systems 70.28 71.34CA168 Digital World 63.81 65.26ES125 Social&Personal Dev with Communication Skills 67.00 66.46HR101 Psychology in Organisations 59.43 63.32LG101 Introduction to Law 53.33 54.85LG116 Introduction to Politics 45.68 44.85LG127 Business Law 60.57 61.82MS136 Mathematics for Economics and Business 60.78 69.35SS103 Physiology for Health Sciences 55.27 57.03Overall Dff in all modules 58.36 61.22
Average scores for participants are higher in 8 of the 10 modules analysed, significantly higher in BE101, and CA103
Module Average Performance Participants vs. Non-Participants
![Page 40: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/40.jpg)
Measuring the Flipping effect
Patrick DoyleMark GlynnEveyn Kelleher
@glynnmark
![Page 41: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/41.jpg)
Assessment Challenge
![Page 42: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/42.jpg)
Logistics
• 200+ students• 4 assignments each• 5 minutes per
assignment• 10 lecturers• 2 weeks of assessment
![Page 43: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/43.jpg)
Marking guide
![Page 44: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/44.jpg)
Related research
Comparing students who watched versus not watched video oneComparing Means [ t-test assuming unequal variances (heteroscedastic) ]
Descriptive StatisticsVAR Sample size Mean Variance
Didn't watch 84 51.86905 691.58505Watched 102 63.15686 576.74743
Two-tailed distribution
p-level 0.00284 t Critical Value (5%) 1.97402
One-tailed distribution
p-level 0.00142 t Critical Value (5%) 1.65387
![Page 45: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/45.jpg)
Discussion
What would you like to measure?
![Page 46: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/46.jpg)
Selectively Analyzing your Course data?
@glynnmark@drjaneholland
Dr Jane Holland, RCSIEric Clarke, RCSIDr Mark Glynn, DCUDr Evelyn Kelleher, DCU
![Page 47: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/47.jpg)
Constructive Alignment
Learning Outcomes
![Page 48: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/48.jpg)
Particulars
• Attendance– Tutorials– labs
• Moodle logs• Defined times• Assessment results
![Page 49: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/49.jpg)
Excel results Video tracking
Zero One Two Three Four Five Six Seven0%
10%
20%
30%
40%
50%
60%
70% What students watched "x" amount of videos
Watched
Watched before
![Page 50: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/50.jpg)
All activities
![Page 51: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/51.jpg)
One activity in particular
![Page 52: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/52.jpg)
Multiple activities
![Page 53: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/53.jpg)
Health warning
![Page 54: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/54.jpg)
Questions and discussion…
![Page 55: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/55.jpg)
Talking to one another
LMS
SRS
CMS
Timetable
Wifi
Library
![Page 56: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/56.jpg)
Databridge
MITM
Course Databa
se
Timetable
ePortfolio
Wifi
LMS
Library
SRS
![Page 57: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/57.jpg)
Additional slides
![Page 58: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/58.jpg)
Building classifiers for each week/each module
Training DataTesting
![Page 59: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/59.jpg)
Notes on model confidence• Y axis is confidence in AUC ROC (not probability)• X axis is time in weeks• 0.5 or below is a poor result• Most Modules start at 0.5 when we don't have much
information• 0.6 is acceptable, 0.7 is really good (for this task)• The model should increase in confidence over time• Even if confidence overall increases, due to randomness
the confidence may go up and down• It should trend upwards to be a valid model and viable
module choice
![Page 60: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/60.jpg)
BExxx: Intro to Cell Biology
Results / year = ~300Pass rate = 0.86
![Page 61: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/61.jpg)
BExxx
![Page 62: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/62.jpg)
SSxx: Health Sciences
Results / Year = ~150Pass rate = 0.92
![Page 63: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/63.jpg)
MSxxx
![Page 64: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/64.jpg)
LGxxx
![Page 65: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/65.jpg)
HR101
![Page 66: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/66.jpg)
CAxxx
![Page 67: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/67.jpg)
Some unusable modulesModules where the ROC AUC increases slowly (e.g stays below 0.6) e.g. PS122
![Page 68: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/68.jpg)
Timescale for Rollout
• Still some issues on Moodle access log data transfer to be resolved
• Still have to resolve student name / email address / Moodle ID / student number
• Still to resolve timing of when we can get new registration data, updates to registrations (late registrations, change of module, change of course, etc.) …
• Should we get new, “clean” data each week ?
![Page 69: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/69.jpg)
Why did you take part?• The majority of students
wanted to learn/monitor their performance
• Many others were curious
• Some were interested in the Research aspect
• Some were just following advice
• Others were indifferent
![Page 70: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/70.jpg)
How easy was it to understand the information in the emails ?(1= not at all easy, 5 = extremely easy)
• Average 3.97 (SD= 1.07)
• Very few had comments to make (19/133)– Most who commented wanted more
detail.
![Page 71: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/71.jpg)
Week 3
Training DataTesting
![Page 72: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/72.jpg)
Week 4
Training DataTesting
![Page 73: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/73.jpg)
Week 5
Training DataTesting
![Page 74: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/74.jpg)
Week 6
Training DataTesting
![Page 75: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/75.jpg)
Week 7
Training DataTesting
![Page 76: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/76.jpg)
Week 8
Training DataTesting
![Page 77: Analytics - Presentation in DkIT](https://reader036.vdocuments.us/reader036/viewer/2022062523/58f1a3a81a28ab703a8b45b3/html5/thumbnails/77.jpg)
Week 9
Training DataTesting