Using learning analytics to uncover learning strategies

Dragan Gašević @dgasevic

Shape of Educational Data Meeting April 7, 2016, Fairfax, VA

Growing role of technology to flip classrooms

Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 201319030.

How do students study with technology?

BACKGROUND

Categorization Deep and surface approaches to learning

Trigwell, K., & Prosser, M. (1991). Relating approaches to study and quality of learning outcomes at the course level. British Journal of Educational Psychology, 61(3), 265-275.

Poor choices of learning tactics and strategies

Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual review of psychology, 64, 417-444.

Significant role of instructions on approaches to learning

Trigwell, K., Prosser, M., & Waterhouse, F. (1999). Relations between teachers’ approaches to teaching and students’ approaches to learning. Higher Education, 37(1), 57–70.

Role of course design

To prompt active engagement and challenge higher order thinking

Bryson, C., & Hand, L. (2007). The role of engagement in inspiring teaching and learning. Innovations in Education and Teaching International, 44(4), 349–362.

Student profiling

Unsupervised approaches

Lust, G., Elen, J., & Clarebout, G. (2013). Students’ tool-use within a web enhanced course: Explanatory mechanisms of students’ tool-use pattern. Computers in Human Behavior, 29(5).

Sequences of activities

Sequence or process mining, HMMs, etc.

Reimann, P., Markauskaite, L., Bannert, M. (2014). e-Research and learning theory: What do sequence and process mining methods contribute? British Journal of Educational Technology, 45(3), 528-540.

What learning strategies do students follow

while using technology?

Do learning strategies of students change over time

while using technology?

FLIPPED CLASSROOM STUDY

Study context

Freshman course in computer systems at USyd

Enrolment: ~300 students

Assessment: midterm + final + project

Flipped learning design

Redesigned lecture – an active learning session requiring students’ preparation

Flipped learning design

Videos with multiple-choice questions (MCQs)

Documents with embedded MCQs

Problem (exercise) sequences

Exploratory sequence analysis

[1] (CONTENT_ACCESS,3)

[2] (EXE_IN,3)-(EXE_CO,1)-(EXE_IN,1)-(EXE_CO,1)-(EXE_IN,2)

[3] (CONTENT_ACCESS,3)-(EXE_IN,4)

[4] (MC_EVAL,4)

[5] (EXE_IN,5)-(EXE_CO,1)-(EXE_IN,3)-(EXE_CO,1)-(EXE_IN,2)-(EXE_CO,1)-(EXE_IN,9)-(EXE_CO,4)-(EXE_IN,4)-(EXE_CO,1)-(EXE_IN,2)-(EXE_CO,2)-(EXE_IN,3)-(EXE_CO,3)-(EXE_IN,1)-(EXE_CO,2)-(EXE_IN,1)

[6] (CONTENT_ACCESS,2)

Gabadinho, A., Ritschard, G., Müller, N.S. & Studer, M. (2011). Analyzing and visualizing state sequences in R with TraMineR, Journal of Statistical Software, 40(4), 1-37.

Agglomerative hierarchical clustering of sequences based on Ward’s algorithm and Levenshtein’s edit distance

Clusters of learning sequences

Pattern/strategy 1 (1354, 11.93%): focus on formative assessment, followed by metacognitive evaluation activities

Pattern/strategy 2 (4736, 41.72%): focus on summative assessment with indicators of trial-and-error learning

Clusters of learning sequences

Pattern/strategy 3 (3228, 28.44%): focus on reading lecture materials with tiny fraction of formative assessment Pattern/strategy 4 (2033, 17.91%): focus on the course videos, with not negligible amount of formative assessment activities; small fraction of metacognitive evaluation activities at the beginning of the learning sessions

Student clustering based on sequence clusters

All the cluster pairs, except for the 1-2 pair, are significantly different (even after applying the FDR correction for multiple testing) in terms of both midterm and final exam scores

Intensive/adaptive Strategic/effective Selective/efficiency Minimalist

Changes in learning strategy

Feature Feature description

MCQ.TOT.FACT Discretized count of completed formative assessment items (MCQs)

MCQ.PERC.CO.FACT Discretized percentage of correctly solved MCQs

EXC.TOT.FACT Discretized count of completed summative assessment items (exercises)

EXC.PERC.CO Discretized percentage of correctly solved exercises

VID.TOT.FACT Discretized count of play and pause video events

MCQ.SH.TOT.FACT Discretized count of requests for answers on formative MCQs

TG.DENS.FACT Discretized transition graph density

MC.EVAL.FACT Discretized count of dashboard and Hall of Fame views

CONTENT.ACCESS.FACT Discretize count of accesses to the lecture content pages

Changes in learning strategy

State Short description Correspondence to sequence-based student clusters

1 Low activity level; focus on lecture materials and summative assessment

Minimalists

2 High activity level; students are engaged with all the preparation activities and are experimenting with different learning strategies

Intensive / adaptive

3 Disengaged -

4 Moderate activity level; similar to state 2 in term of engagement and the diversity of learning strategies, but with lower activity level

Strategic / effective

5 Focus on summative assessment; low engagement with lecture materials and very rarely with the course videos; skipping formative assessment

Selective / efficiency-oriented

Changes in learning strategy

CONCLUSION

Learning approaches consistent with the literature

Effective vs ineffective learning strategies

Please, don’t confuse this with learning styles

Pet peeve!

Students don’t follow the same approach all the time

Process nature of learning - beyond coding and counting -

van der Aalst, W. (2012). Process mining: Overview and opportunities. ACM Transactions on Management Information Systems (TMIS), 3(2), 7.

Critical role of course design and contextual variables

Critical role of course design

Critical role of course design

Trace data about internal conditions needed

Thanks you!