the role of learning analytics: a personal journey… 5/22/2013 oded meyer department of mathematics...
TRANSCRIPT
![Page 1: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/1.jpg)
The Role of Learning Analytics: A personal Journey…
5/22/2013
Oded Meyer
Department of Mathematics and Statistics
Georgetown University
![Page 2: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/2.jpg)
Some background about myself…..
![Page 3: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/3.jpg)
![Page 4: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/4.jpg)
![Page 5: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/5.jpg)
![Page 6: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/6.jpg)
![Page 7: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/7.jpg)
![Page 8: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/8.jpg)
![Page 9: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/9.jpg)
![Page 10: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/10.jpg)
![Page 11: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/11.jpg)
Learning Analytics
Data driven approach for the purpose
of understanding and improvinglearning and the and the
environmentin which it occurs.
![Page 12: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/12.jpg)
Carnegie Mellon Open Learning Initiative (OLI)
Scientifically-based online learning environments based on the integration of technology and the science of learning with teaching. OLI is designed to simultaneously improve learning and facilitate learning research.
![Page 13: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/13.jpg)
The OLI Statistics Course
![Page 14: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/14.jpg)
Educational Mission of Funder (2002) (The William and Flora Hewlett Foundation)
Provide open access to high quality post-secondary education and educational materials to those who otherwise would be excluded due to:– Geographical constraints– Financial difficulties– Social barriers
To meet this goal:– A complete stand-alone web-based introductory
statistics course.– openly and freely available to individual learners online.
![Page 15: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/15.jpg)
![Page 16: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/16.jpg)
Important form of analytics: The Science of Learning
General:
• Make the structure and “big picture” salient
![Page 17: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/17.jpg)
Learning Science Principles - continued
• Immediate and targeted feedback students achieve desired level of performance faster.
![Page 18: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/18.jpg)
![Page 19: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/19.jpg)
Science of Learning - continued
Discipline specific principles:
• Hands-on activities• Use real data• De-emphasize calculations.
![Page 20: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/20.jpg)
Start Analytics Early in the Design process
After one module :
• Qualitative feedback• Observe students• Talk-allowed protocols
![Page 21: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/21.jpg)
Instructor feedback loop analytics
• Feedback to the instructor about students’ learning Learning Dashboard
- Presents the instructor with a measure of student learning for each learning objective.
- More detailed information:• Class’s learning of sub-objectives• Learning of individual students• Common misconceptions
![Page 22: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/22.jpg)
Learning Dashboard Team led by Dr. Marsha Lovett
![Page 23: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/23.jpg)
![Page 24: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/24.jpg)
![Page 25: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/25.jpg)
![Page 26: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/26.jpg)
Instructor feedback loop analytics
Key:• Clearly defined learning objectives.• Tying each activity to a learning objective.
Benefits:• Can reveal misconceptions• Impacts how your spend class time• Can reveal “expert blind spots”
![Page 27: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/27.jpg)
Students who achieved proficiency in finding median did poorly in the following assessment item tied to the same sub-skill:
![Page 28: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/28.jpg)
Larger Scale Analytics:
Assessing the Effectiveness of the Course When used in the Blended (Hybrid) Mode
![Page 29: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/29.jpg)
![Page 30: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/30.jpg)
![Page 31: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/31.jpg)
![Page 32: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/32.jpg)
Larger Scale Analytics: Assessing the Effectiveness of the Course
The Hewlett Foundation’s “AccelerateLearning Challenge”:
Can students using the OLI course in the blended mode learn the same material as
they would in the traditional course in shorter time and still have equal or better
learning gains.
![Page 33: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/33.jpg)
Three accelerated Studies
#1 Small class, expert instructor (2007)
#2 Replication with larger class (2009)With retention follow-up 4+ months later
#3 Replication with new instructor (2010)
Experienced statistics instructorNew to OLI Statistics course and hybrid
mode
![Page 34: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/34.jpg)
Study 1: Method~180
studentsenrolled
68 volunteers for special
section
44 students,traditional
control condition
24 students,adaptive/
accelerated condition
![Page 35: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/35.jpg)
Adaptive/Accelerated vs. Traditional
Two 50-minute classes/wk
Eight weeks of instruction
Homework: complete OLI activities on a schedule
Tests: Three in-class exams, final exam, and CAOS test
Four 50-minute classes/wk
Fifteen weeks of instruction
Homework: read textbook & complete problem sets
Tests: Three in-class exams, final exam, and CAOS test
<<
?
=
Same content but different kind of instruction
![Page 36: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/36.jpg)
Dependent Measure
CAOS = Comprehensive Assessment of Outcomes in a First Statistics course (delMas, Garfield, Ooms, Chance, 2006)
– Forty multiple-choice items measuring students’ “conceptual understanding of important statistical ideas”
– Content validity – positive evaluation by 18 content experts
– Reliability – high internal consistency– Aligned with content of course (both sections)– Administered as a pre/posttest
![Page 37: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/37.jpg)
Study 1: CAOS Test Results
Adaptive/Accelerated group gained more (18% vs 3%)pre/post on CAOS than did Traditional Control, p < .01.
Chance
![Page 38: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/38.jpg)
These analytics got the attention of education leaders in the U.S. who are facing the “cost disease” in higher ed.
William Bowen (former President of Princeton)replicated our study in the “Interactive Learning Online at Public Universities” study.
The study further indicated that blended learning offers the potential of more economical and rapid pathways to mastery.
![Page 39: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/39.jpg)
Analytics on Students’ Learning Habits
• Students in both groups recruited to complete time-logs
• Self-report for both groups• Analogous point in the course (2/3 through)• Six consecutive days: Wednesday - Monday
![Page 40: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/40.jpg)
Study 1: Time Spent Outside of Class
No significant difference between groups in the time students spent on Statistics outside of class
![Page 41: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/41.jpg)
Study 2: Replication & Extension
• Same method, same procedure, same instructor
• Larger class (52 students in Adaptive / Accelerated)
• Follow-up retention study conducted 4+ months later
![Page 42: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/42.jpg)
Study 2: CAOS Test Results
Adaptive/Accelerated group gained more pre/post on CAOS than did Traditional Control, p < .01.
Chance
![Page 43: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/43.jpg)
Using Analytics to Assess Retention
Jan Feb Mar Apr May Jun Jul Aug Sep Oct
Follow-up Begins
Adapt/Acc EndsTrad’l Ends
Adapt/Acc Delay (13 Students)
Trad Delay (14 Students)
![Page 44: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/44.jpg)
Study 2, Retention: Re-taking CAOS
At 6-month delay, Adaptive/Accelerated group scored higher on CAOS than Traditional Control, p < .01.
Chance
![Page 45: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/45.jpg)
Study 3: Further Replication & Extension
• Same method, same procedure
• New instructor• Not involved in development of OLI course• New to OLI statistics and hybrid teaching mode• Instructor held constant for both Adapt/Acc and
Control conditions
• Larger class (40 students in Adaptive / Accelerated)
![Page 46: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/46.jpg)
Study 3: CAOS Test Results
0
10
20
30
40
50
60
70
Adapt/Acc Trad Control
Pretest
Posttest
Adaptive/Accelerated group gained more pre/post on CAOS than did Traditional Control, p < .01.
Chance
![Page 47: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/47.jpg)
Current and Future Analytics
• Continued “gap analysis”.
• Better alignment between learning objectives/sub-objectives and activities.
• Student Dashboard provide learners with insight into their own learning habits and can give recommendations for improvement.
• Learning-facing analytics allows learners to compare their own performance against an anonymous summary if their course peers
![Page 48: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/48.jpg)
Lessons Learned…
1. Pedagogy must drive technology and not the other way.
2. Developing online materials is a collaborative effort.
3. Developing online materials is an iterative process.
4. Steep learning curve.5. Growth as a teacher.
![Page 49: The Role of Learning Analytics: A personal Journey… 5/22/2013 Oded Meyer Department of Mathematics and Statistics Georgetown University](https://reader035.vdocuments.us/reader035/viewer/2022081513/56649cb05503460f94974fad/html5/thumbnails/49.jpg)
“Improvement in post-secondary education will require converting teaching from a ‘solo sport’ to a community-based research activity”
Herbert Simon, Last Lecture Series, Carnegie Mellon, 1998