follow the successful crowd: raising mooc completion rates through social comparison at scale

FOLLOW THE SUCCESSFUL CROWDRAISING MOOC COMPLETION RATES THROUGH SOCIAL COMPARISON AT SCALE

DAN DAVIS IOANA JIVET RENÉ KIZILCEC LAMBDA-LAB

GUANLIANG CHEN CLAUDIA HAUFF GEERT-JAN HOUBEN

MOOC LEARNING

FEEDBACK

SOCIAL CUES SELF-REGULATED LEARNING METACOGNITION

MOTIVATION

OUTLINE 2. EXPERIMENTS3. CULTURE4. CONCLUSION

1. DESIGN

1. DESIGNFEEDBACK · SOCIAL COMPARISON

SOCIAL COMPARISON THEORYEstablish social and personal worth by comparing ourselves to others (Festinger, 1954)

Comparison with slightly-better-performing peers has a beneficial effect on students’ grades (Huguet, 2001)

ROLE MODELSlearners who earned a passing grade in the previous edition of the MOOC

FEEDBACK SYSTEM DESIGN

DESIGN GOALSprovide feedback on behavior extracted from edX’s log traces afford learners the ability to intuitively understand and explore the information presented (Kay) show learners specific ways to take action and change their behavior

Traceable

Scrutable

Actionable

Learning tracker

Average graduate You kAverage graduate next week

Time on theplatform

Time watchingvideos

Fraction of timespent watchingvideos while onthe platform

Videos watched

Quiz answerssubmitted

Timeliness ofquiz answersubmission

FEEDBACK SYSTEM DESIGN

1. Embodied representation of multiple metrics 2. Numerous indicators displayed in a small space 3. Easily comparable

ITERATIVE DESIGN1. Interactivity 2. Planning ahead 3. Framing

Framing

Feedback Dimension

Planning Ahead

Interactive

2. EXPERIMENTSRESEARCH QUESTIONS · RESULTS

RQ1 Does social comparison-based learner feedback lead to increased achievement and increased self-regulatory behavior for MOOC learners?

Water Treatment

Urban Sewage

Responsible Innovation

Pre-Calculus

11k

8k

2k

12k

3.5%

3.8%

3.6%

1.7%

42

272

79

327

MOOC ENROLLED PASS QUIZQUESTIONSRATE

5,460

5,483

—

Control

Treatment 1

Treatment 2

4,038

4,099

—

1,184

1,168

—

4,142

4,087

4,065

WaterX SewageX BusinessX CalcX

ALL ENROLLED

WaterX SewageX BusinessX CalcX

1,268

1,251

—

Control

Treatment 1

Treatment 2

771

746

—

164

160

—

1,150

1,147

1,118

ACTIVE: >5 MIN IN COURSE

RESULTSCOMPLETION · ENGAGEMENT · PRIOR EDU

COURSE COMPLETION

WaterXC 1,268 160

MOOC COND. N # PASS PASS RATE

T 1,251 18812.6%15.0%

UrbanXC 771 136T 746 165

17.6%22.1%

BusinessXC 164 46T 160 54

28.0%33.8%

OVERALLC 2,203 342T 2,157 407

15.5%18.9%

WaterXC 1,268 160

MOOC COND. N # PASS PASS RATE

T 1,251 18812.6%15.0%

UrbanXC 771 136T 746 165

17.6%22.1%

BusinessXC 164 46T 160 54

28.0%33.8%

OVERALLC 2,203 342T 2,157 407

15.5%18.9% **

WaterX UrbanX BusinessX Overall

Control

Treatment

10%

20%

30%

COURSE COMPLETION


Providing learners a comparison of their own behavior to that of previously successful peers will increase learner achievement.

H1.1


Providing learners a comparison of their own behavior to that of previously successful peers will increase learner achievement and engagement

H1.1

Learners will change the aspects of their behavior that the Feedback System makes them aware of.

H1.2

X

RQ2 Which learners benefit most from the Feedback System?

PRIOR EDUCATION

WaterXHigh 13.2

MOOC PRIOR

Low 11.815.711.5

UrbanX17.416.4

21.314.0

BusinessX23.721.4

29.122.8

OVERALL16.314.4

19.513.6

HighLowHighLowHighLow

EDU

0.150.16

0.040.66

0.040.78

<0.010.36

CTRL TREAT pGRADE GRADE

WaterXHigh 13.2

MOOC PRIOR CTRL TREAT

11.815.711.5

UrbanX17.416.4

21.314.0

BusinessX23.721.4

29.122.8

OVERALL16.314.4

19.513.6

High

High

High

EDU p

0.150.16

0.040.66

0.040.78

<0.010.36

<

<

<

<

GRADE GRADE

Low

Low

Low

Low


A: HIGHLY EDUCATED LEARNERS

3. CULTURE*FRAMING · RESULTS

* PRE-REGISTERED osf.io/ys6au/

RQ3Does feedback framed in line with a learner’s cultural context make it more effective?

Framing

Feedback Dimension

Planning Ahead

Interactive

ON TRACK

BEHIND

Framing

Feedback Dimension

Planning Ahead

Interactive

CULTUREHOFSTEDE INDIVIDUALISM

INDIVIDUALISM

INDIVIDUAL AS AN INDEPENDENT ACTOR WITH LOOSE SOCIAL RELATIONS

COLLECTIVISM

CLOSELY KNIT SOCIAL RELATIONS AND GREAT CARE FOR THE WELL-BEING OF ALL

INDIVIDUALISMCOLLECTIVISM

CULTUREHOFSTEDE INDIVIDUALISM

GELFAND TIGHTNESS

HAVE MANY STRONGLY ENFORCED RULES AND LITTLE TOLERANCE FOR DEVIANCE

TIGHTLOOSE

HAVE FEW STRONGLY ENFORCED RULES AND GREATER TOLERANCE FOR DEVIANCE

TIGHTLOOSE


Feedback in line with a learner’s cultural individualism index will increase its impact on the learner

H3.1

X


Feedback in line with a learner’s cultural individualism index will increase its impact on the learner

H3.1

Feedback in alignment with a learner’s cultural tightness will increase its impact on the learner

H3.2

X

WaterX

MOOC PASS RATE

13.8%

UrbanX 19.8%

BusinessX 31%

CalcX 5%

RAISE ACTIVITY THRESHOLD5 MINUTES -> 1 HOUR

From N = 3,415 to N = 658

WaterX

MOOC PASS RATE

13.8%

UrbanX 19.8%

BusinessX 31%

CalcX 16%

INDIVIDUALIST FRAMING+ Activity Threshold

Increased completion rates across cultural contexts

Control: 12.8%Individualist Framing: 19.9%

55% Increase (t = 2.02, p = 0.04, N = 441)

INDIVIDUALIST FRAMING+ Activity Threshold

Increased completion rates across cultural contexts

Control: 12.8%Individualist Framing: 19.9%

55% Increase (t = 2.02, p = 0.04, N = 441)

Increased completion rates especially in tight culturesControl: 12.1%Individualist Framing: 36.3%

300% Increase (t = 2.07, p = 0.04, N = 55)

5. CONCLUSIONWHAT’S NEXT

1. REAL-TIME 2. FRAMING 3. PERSONAS

THANK YOU

BIT.LY/WIS-LEARNING-ANALYTICS

LAMBDA-LAB

DAN DAVIS IOANA JIVET RENÉ KIZILCEC

GUANLIANG CHEN CLAUDIA HAUFF GEERT-JAN HOUBEN

LOW CULTURAL TIGHTNESS PREDICTS COMPLETION

p < 0.02

Control 1,150 45

CONDITION N # PASS PASS RATE

3.9%

Indiv. - Promotion 1,147 62 5.4%

Collect. - Prevent. 1,118 51 4.5%

OVERALL EFFECT

follow the successful crowd: raising mooc completion rates through social comparison at scale

Education