learning at scale: using research to improve learning practices and technology for teaching math
TRANSCRIPT
Learning at ScaleUsing Research To Improve Learning Practices
and Technology for Teaching Math
Maria H. Andersen, Ph.D.Chief Troublemaker, Edge of Learning
Math Faculty, Westminster College
What have we learned?
10 Lessons from Learning at Scale
Stand up if you know what a MOOC is.
Stay standing if you have ever signed up for a MOOC.
Stay standing if you have ever finished a MOOC.
HarvardX and MITx: Four Years of Open Online CoursesAuthors: I. Chuang, A. HoPublished: 2016Study size: 4.4 million students across 290 MOOCs
If a MOOC held 100 students …
(Chuang & Ho, 2016)
(Chuang & Ho, 2016)
How many times a year do you get a chance to start over?
1. We need to improve findability.
Learners are in the maze, and they can’t see
the big picture.
Many of you have been in a maze like what our students experience.
Wouldn’t it be nice to have a transporter to get right to where you want to go?
Why don’t students seem to be able to find information that is clearly printed in the syllabus?
The impact of findability on student motivation, self-efficacy, and perceptions of online course qualityAuthors: B. Simunich, D. Robins & V. KellyPublished: 2015Study size: n=81, students had to complete 7 tasks, navigating either a
well-constructed course (according to QM) or a “broken” course
What else can we do to help learners navigate
this maze?
Help students to build a physical map to anchor memories as they learn.
7.00x Introduction to Biology: The Secret of Life MITx on EdX Course Report - 2013 SpringAuthors: D. Seaton, J. Reich, S. Nesterko, et al.Published: 2014Study size: 38k signups, 3K certified
(Seaton et al, 2014)
6.00x Introduction to Computer Science and Programming MITx on edX – 2012 FallAuthors: S. Features, P. Grimson, J. GuttagPublished: 2014Study size: 84.5k signups, 5.7K certified
14.73x The Challenges of Global Poverty MITx on edX – 2013 SpringAuthors: P. Black, P. WhitePublished: 2014Study size: 40k signups, 4.6K certified
(Seaton et al, 2014)
Introduction to Biology
(Features et al, 2014)
Introduction to Computer Science and Programming
(Black & White, 2014)
The Challenges of Global Poverty
Examining Engagement: Analysing Learner Subpopulations in MOOCsAuthors: R. Ferguson, D. ClowPublished: 2015Study size: 34k signups, 7k full participants, 4 MOOCs
(Ferguson & Clow, 2015)
HarvardX and MITx: Four Years of Open Online CoursesAuthors: I. Chuang, A. HoPublished: 2016Study size: 4.4 million students across 290 MOOCs
Find Calculus.
(Chuang & Ho, 2016)
2. Students need to be coerced into more frequent engagement.
Are you ever surprised by things you can find on the
Internet?Stand up if you know who
the Borg are.Stay standing if you’ve ever done an internet search for a
cat in a Borg costume.
3. Use an online math
learning system.
Traditional Course
Online homework (quantity and difficulty adapts to mastery levels of student)
Online homework (all students do same problems)
< <
A hierarchy of online math learning based on principles from learning research
Online homework (adapts to mastery levels of student and learner profile or preferences)
<
Supports each students’ pace
Interaction around content
Frequent accountability
Immediate feedback
Learning is not a spectator sport: doing is better than watching for learning from a MOOCAuthors: K. Koedinger, J. Kim, J. Jia, et alPublished: 2015Study size: 27k registrants, 1k completers, “Introduction to Psychology as a Science” delivered through Coursera and OLI platform
Our digital lives are with us everywhere, our learning should be too.
Stand up if you teach online or rely heavily on digital learning materials.
Stay standing if you think it’s easy to get students to
participate in these.
Stay standing if you use discussion boards with your students.
4. Get students to participate in their learning
community.
1-9-90 Rule
1% creators9% commenters/sharers90% lurkers
Comments in MOOCs: who is doing the talking and does it help?Authors: B. Swinnerton, S. Hotchkiss & N.P. MorrisPublished: 2017Study size: 25k active learners, 8k commenters, 5k students who completed a pre-course survey (roughly half commenters and half non-commenters)
Predicting Student Retention in Massive Open Online Courses using Hidden Markov ModelsAuthors: G. BalakrishnanPublished: 2013Study size: 30k enrolled students in “Software as a Service” MOOC
(Balakrishnan, 2013)
Commenting in discussion boards
(Balakrishnan, 2013)
Commenting in discussion boards
(Balakrishnan, 2013)
Lurking in discussion boards
Use announcements
to showcase great discussion posts.
5. Use personas to strategize
about interventions.
Using Data Mining to Differentiate Instruction in College AlgebraAuthors: R. ManspeakerPublished: 2011Study size: 524 students in College Algebra using digital homework,
over 2 semesters
(Manspeaker, 2011)
Overachievers (OA)
Characteristics What do they struggle with?• see mathematics as useful • do not see how the class
specifically applies to their lives• high grades in math
• non-standard problems• problems not found on
homework or worked in class
Underachievers (UA)
Characteristics What do they struggle with?• Intelligent and well-
prepared• Bored and frustrated with
the course• Do reasonably well on the
first exam
• Procedural and non-standard problems• Get low scores in most
aspects of the course• High risk for dropping out
(Manspeaker, 2011)
“Employees” (E)
Characteristics What do they struggle with?• Feel the class is a menial job
for which they get “paid” with a passing grade• Dislike math• Learn through memorizing• Excel at procedural
problems
• Problems involving creative thinking or independent thought• Attendance (starts strong,
then drops)
(Manspeaker, 2011)
(Manspeaker, 2011)
Rote Memorizers (RM)
Characteristics What do they struggle with?• Rely on memorizing to get
through math• Negative views of math
which will get worse over time• Poorest scores
• Emporium-style courses• Everything
Sisyphean Strivers (SS)
Characteristics What do they struggle with?• Excellent Attendance• Do well on written
homework• Have high understanding of
math if interviewed• Do very poorly on exams
• Non-standard questions• Application problems• Exams• Online homework
(Manspeaker, 2011)
Stand up if students drop your classes.
Stay standing if the digital systems you use make it easy to see which
students you should intervene with.
Stay standing if you have strategies in place to try to
intervene before disengagement.
6. Anticipate and catch students as they fall off the path to success.
Community Insights: Emerging Benchmarks & Student Success Trends from across the CivitasAuthors: Civitas LearningPublished: 2016Study size: 2 million learners, 55 colleges
LMS activity in the first 14 days
LMS activity in the first 14 days
Persistence: 92%
(or more)
Persistence: 76%
LMS activity in the first 14 days
Persistence: 47%
LMS activity in the first 14 days
Using Bayesian Learning to Classify College Algebra Students by Understanding in Real-TimeAuthors: A. CousinoPublished: 2013Study size: 505 students over 2 semesters
(Cousino, 2013)
Examining Engagement: Analysing Learner Subpopulations in MOOCsAuthors: R. Ferguson, D. ClowPublished: 2015Study size: 34k signups, 7k full participants, 4 MOOCs
(Ferguson & Clow, 2015)
Samplers Visit, but only briefly (commented in some courses)Strong Starters Complete the first week, but then dropout
Returners Made it through 2 weeks, but then dropout
Midway Dropouts Dropped out around assessment 3 or 4
Nearly There’s Made it through ¾ of the course, then dropped
Late Completers Completed course, but submitted things late
Keen Completers Completed course, mostly on time (>80%)
(Ferguson & Clow, 2015)
(Graph generated from data in Ferguson & Clow, 2015)
Deconstructing Disengagement: Analyzing Learner Subpopulations in Massive Open Online CoursesAuthors: R. Kizilcec, C. Piech, E. SchneiderPublished: 2013Study size: 94k active learners in 3 computer science MOOCs
7. Courses need a catch-up option.
Why don’t we offer 1-2 week
”slipped schedule” courses?
This might mean the
courses have to be offered on 12-week schedules.
8. Tweak instructional videos.
How Video Production Affects Student Engagement: An Empirical Study of MOOC VideosAuthors: P. Guo, J. Kim, R. RubinPublished: 2014Study size: 6.9 million video watching sessions across four
courses on the edX MOOC platform
46% 33%
(Guo et al, 2014)
(Guo et al, 2014)
(Guo et al, 2014)
Planned vs. Chopped
9. Don’t aim for perfect. Let students get stuck a little more.
Your click decides your fate : Inferring Information Processing and Attrition Behavior from MOOC Video Clickstream Interactions
Authors: T. Sinha, P. Jermann, N. Li et al.Published: 2014Study size: 66k enrolled, 36.5k interacted with videos, there were 48
video lectures (producing 10 Gb of JSON data) from “Functional Programming in Scala” run on Coursera
10 GB of JSON video click dataPlay (Pl) Pause (Pa) SeekFw (Sf)SeekBw (Sb)
ScrollFw (SSf)ScrollBw (SSb)RatechangeFast (Rf)RatechangeSlow (Rs)
PlPaSbPl RfPaPlPa(rewatch) (fast watching)
SSbSbPaPl(clear concept)
(Sinha et al., 2014)
(Sinha et al., 2014)
Information Processing Index (IPI)
(Sinha et al., 2014)
If students’ rewatching behavior changes from low to high IPI, they are 33% less likely to drop out.
Why do we lose students before graduation?
Community Insights: Emerging Benchmarks & Student Success Trends from across the CivitasAuthors: Civitas LearningPublished: 2016Study size: 4 million learners (this part was conducted looking at 2
years of data), 55 colleges
(Civitas, 2016)
Academic Performance is not the Primary Risk to Departure
10. Make sure your courses are have opportunities for curiosity,
challenge, and creativity.
Let’s see what you remember…
1. Improve findability.
2. Coerce students into more frequent engagement.
3. Use an online math
system.
4. Get students to participate in their learning community.
5. Use personas to strategize
about interventions.
6. Anticipate and catch
students as they fall off the path to success.
7. Courses need a catch-up option.
8. Tweak Instructional
Videos.
9. Let students get stuck a little more.
10. Make sure your courses are have opportunities for curiosity,
challenge, and creativity.
Cool. You remember
everything. My work here is done.
