engaging with massive online coursesashton/slides/moocengagement-… · we studied 6 coursera...
TRANSCRIPT
![Page 1: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/1.jpg)
Engaging with Massive Online Courses
Ashton Anderson, Daniel Huttenlocher, Jon Kleinberg, Jure Leskovec
Monday, April 7, 14
![Page 2: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/2.jpg)
massive online courses
over 1,000 moocs offered
across many platforms
10 million students and counting
2
Monday, April 7, 14
![Page 3: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/3.jpg)
massive online courses
potential to revolutionize higher education
...and our understanding of how people learn
3
Monday, April 7, 14
![Page 4: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/4.jpg)
massive online courses
but first:
how do students engage with moocs?
can engagement be incentivized?
4
Monday, April 7, 14
![Page 5: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/5.jpg)
the data
we studied 6 coursera classes:
3 machine learning (Andrew Ng)3 probabilistic graphical models (Daphne Koller)
(Thanks to Coursera and the Stanford Lytics Group for sharing the data with us!)5
Monday, April 7, 14
![Page 6: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/6.jpg)
the data
Basic course statistics
6
Monday, April 7, 14
![Page 7: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/7.jpg)
engaging with massive online courses
1. participation
2. performance
3. interaction
4. intervention
7
Monday, April 7, 14
![Page 8: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/8.jpg)
1. participation
2. performance
3. interaction
4. intervention
engaging with massive online courses
8
Monday, April 7, 14
![Page 9: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/9.jpg)
engagement styles
are different students using moocs differently?
can we classify students by their engagement styles?
9
Monday, April 7, 14
![Page 10: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/10.jpg)
engagement styles
Histogram over students’ assignment fractions
10
Monday, April 7, 14
![Page 11: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/11.jpg)
engagement styles
Same for all courses 11
Monday, April 7, 14
![Page 12: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/12.jpg)
engagement styles
Solvers
All-roundersViewers + Collectors
Not many actions: Bystanders
12
Monday, April 7, 14
![Page 13: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/13.jpg)
engagement styles
Number (fraction) of students of each style
13
Monday, April 7, 14
![Page 14: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/14.jpg)
engagement styles
eventual engagement style vs. registration time14
Monday, April 7, 14
![Page 15: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/15.jpg)
engagement styles
we find a large fraction of archaelogists, students who register after the class ends
15
Monday, April 7, 14
![Page 16: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/16.jpg)
engagement styles
we find a large fraction of archaelogists, students who register after the class ends
16
Monday, April 7, 14
![Page 17: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/17.jpg)
1. participation
2. performance
3. interaction
4. intervention
engaging with massive online courses
17
Monday, April 7, 14
![Page 18: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/18.jpg)
performance
18
Grades received
Monday, April 7, 14
![Page 19: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/19.jpg)
performance
19
Distribution of activities for high-achievers
Monday, April 7, 14
![Page 20: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/20.jpg)
1. participation
2. performance
3. interaction
4. intervention
engaging with massive online courses
20
Monday, April 7, 14
![Page 21: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/21.jpg)
interaction
the discussion forums provide a mechanism for students to interact with each other
21
Monday, April 7, 14
![Page 22: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/22.jpg)
interaction
what types of students are active on the forums?
how are the forums being used?
22
Monday, April 7, 14
![Page 23: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/23.jpg)
interaction
given forum usage, which engagement style
all-rounders and viewers make up most of the forums
P (S|F ) :
23
Monday, April 7, 14
![Page 24: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/24.jpg)
interaction
given forum usage, which engagement stylegiven style, likelihood of forum usage
all-rounders and viewers make up most of the forums
90% of all-rounders are on the forums!
P (S|F ) :
P (F |S) :
24
Monday, April 7, 14
![Page 25: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/25.jpg)
interaction
less-active, lower-graded students start threads, more-active, higher-graded students respond
consistent with q&a usage25
Monday, April 7, 14
![Page 26: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/26.jpg)
1. participation
2. performance
3. interaction
4. intervention
engaging with massive online courses
26
Monday, April 7, 14
![Page 27: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/27.jpg)
intervention
can we increase engagement?
27
Monday, April 7, 14
![Page 28: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/28.jpg)
intervention
we designed and implemented a badge system to increase forum engagement on ML3
(Thanks to Pamela Fox and Norian Caporale-Berkowitz for the implementation help!)28
Monday, April 7, 14
![Page 29: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/29.jpg)
Badge name Action Category Criteria Num Badges
Supporter Votes Cumulative Awarded once a user votes 3/15/40/100 times 4 (BSGD)
Reader Reading threads Cumulative Read 10/30/70/200 threads 4 (BSGD)
Good/Great/Awesome/
Incredible ReplyQuality replies Great
AchievementAwarded for contributing a high-quality reply
(5/10/25/100 upvotes) 4 (BSGD)
Good/Great/Awesome/
Incredible ThreadQuality threads Great
AchievementAwarded for contributing a high-quality thread
(5/10/25/100 upvotes) 4 (BSGD)
Contributor “Good” replies Cumulative Great Achievement
Contributing 3/6/10/25 good reply (where good = 3 upvotes) 4 (BSGD)
Conversation Starter “Good” threads Cumulative Great
AchievementContributing 3/6/10/25 good threads
(where good = 3 upvotes) 4 (BSGD)
Community Member Join class First-time Awarded when user joins class (as intro to
badges) 1
Forum Newbie Any First-time Awarded once a user takes any action in the forums 1
Early Bird Vote/Post/Thread Activity Active on forums in first two weeks 1
All-Star Vote/Post/Thread Activity Being active in all weeks 1 29
Monday, April 7, 14
![Page 30: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/30.jpg)
intervention
did the badges have an effect?
implemented badges on ML3, compare observationally with previous runs ML1 and ML2
30
Monday, April 7, 14
![Page 31: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/31.jpg)
intervention
5x more likely to get to 100 votes/reads!31
Monday, April 7, 14
![Page 32: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/32.jpg)
intervention
5x more likely to get to 100 votes/reads!32
Monday, April 7, 14
![Page 33: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/33.jpg)
intervention
no qualitative difference in posts/comments
no badges on these actions!33
Monday, April 7, 14
![Page 34: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/34.jpg)
badgified dimensions ⇒ 5-fold increase in engagement
intervention
unbadgified dimensions ⇒ no qualitative difference
not a true experiment, but very strong observational evidence of badge effect
engagement can be increased in targeted ways! 34
Monday, April 7, 14
![Page 35: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/35.jpg)
intervention, part 2
a true experiment: variation in badge presentation
what gives badges their power?
compare different badge presentations, measure which have strongest effects
35
Monday, April 7, 14
![Page 36: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/36.jpg)
three experiments:
1. top bylines2. thread bylines3. badge ladder
intervention, part 2
36
factorial design (users randomly assigned to 1 of 8 (=23) buckets)
Monday, April 7, 14
![Page 37: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/37.jpg)
intervention, part 2
37
experiment 1: top badge byline
header
Monday, April 7, 14
![Page 38: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/38.jpg)
intervention, part 2
38
experiment 1: top badge byline
Control: Treatment:
Monday, April 7, 14
![Page 39: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/39.jpg)
intervention, part 2
39
experiment 2: thread badge bylines
Posts are annotated with author name
and timestamp
Monday, April 7, 14
![Page 40: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/40.jpg)
intervention, part 2
40
experiment 2: thread badge bylines
Posts are annotated with author name
and timestamp
Monday, April 7, 14
![Page 41: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/41.jpg)
intervention, part 2
41
experiment 2: thread badge bylines
Control:
Treatment:
Monday, April 7, 14
![Page 42: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/42.jpg)
intervention, part 2
42
experiment 3: badge ladder
Control:
Monday, April 7, 14
![Page 43: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/43.jpg)
intervention, part 2
43
experiment 3: badge ladderTreatment:
Monday, April 7, 14
![Page 44: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/44.jpg)
Mann-Whitney rank-sum p-values
Badge ladder most significantExplicit goal-setting helped more than
increased social visibility of badges
intervention, part 2
44
Monday, April 7, 14
![Page 45: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/45.jpg)
conclusion
45
conceptual framework for the quantitative analysis of engagement in moocs
classified users into a taxonomy of engagement styles
designed and implemented a badge system and a randomized experiment that
increased forum engagement
Monday, April 7, 14
![Page 46: Engaging with Massive Online Coursesashton/slides/moocengagement-… · we studied 6 coursera classes: 3 machine learning (Andrew Ng) ... Histogram over students’ assignment fractions](https://reader033.vdocuments.us/reader033/viewer/2022051917/60091b6af9f7de72ae65cf03/html5/thumbnails/46.jpg)
thank you!
46
Monday, April 7, 14