does echo360‐alp analytics predict the relationship ... · • just -in time teaching f2f-class...

#THETA2017

This work is licensed under a Creative Commons Attribution 4.0 International License

Does Echo360‐ALP analytics predict the relationship between 

student behaviours and learning?

Colin Montpetit, Sonya Sabourin

Leveraging the Echo360-ALP to Transform A Large Enrollment

Lecture Classroom into an Active Learning Zone

Colin Montpetit, Ph.D., Assistant Professor

Sonya Sabourin, B.Sc. Honour’s student

Department of Biology, University of Ottawa

2015 TECHTEACH CONFERENCE - UIC

Does Echo360-ALP analytics predict the relationship

between student behaviours and learning?

2015 TECHTEACH CONFERENCE - UIC

Modules (self-paced)• Targeted readings

• Interactive act./Videos

• Reflective act.

• Module quiz OnlineBlackBoard LMSInitial exposure

Summative• Two-Stage Collaborative Exams

(2 midterms + 1 final)

Assessments

Labs• Experiments

• Worksheets, reports

• Quizzes

F2F-Labs

Defined Learning Outcomes, 5E Learning Cycle

Class• Clicker case studies

• Peer-Instruction

• Just-in-Time Teaching

F2F-class

Echo360-ALP

65% 20%

15%

Echo360- Active Learning Platform

2015 TECHTEACH CONFERENCE - UIC

2015 TECHTEACH CONFERENCE - UIC

What is the class usage profile?

2015 TECHTEACH CONFERENCE - UIC

Engagement data provides insight via dashboards

Confusion

Questions

posed

Activity

answers/

participation

Notes

Attendance

Presentation

views

Learning Analytics Database

Lecture capture

2015 TECHTEACH CONFERENCE - UIC

Attendance, Participation, Activity Score

2015 TECHTEACH CONFERENCE - UIC

Note Taking

2015 TECHTEACH CONFERENCE - UIC

Presentation Views

2015 TECHTEACH CONFERENCE - UIC

Video Views

Caroline Petit-Turcotte Winter 2015

447 different students asked at least 1 question

2015 TECHTEACH CONFERENCE - UIC

Questions to the professorBIO1140 – Introduction to Cell Biology

153 different students flagged a slideSOS call out! (flagging slides)BIO1140 – Introduction to Cell Biology

2015 TECHTEACH CONFERENCE - UIC

2015 TECHTEACH CONFERENCE - UIC

Item Analysis – Difficulty Index

2015 TECHTEACH CONFERENCE - UIC

Item Analysis – Discrimination Index

0-10 10_19 20-29 30-39 40-49 50-59 60-69 70-79 80-89 90-99

Pro

po

rtio

n o

f C

las

s (

%)

0

5

10

15

20

Distribution of Learning Gains

2015 TECHTEACH CONFERENCE - UIC

<gains> (ranges)

2011: 30%

2014/16: 60%

Normalized LG = Post – Pre / 100 - pre

What Are Two-Stage Exam?

2015 TECHTEACH CONFERENCE - UIC

Stage 1: IndividualMCQs, T/F, Match

Short Answer

Stage 2: GroupsMCQs, T/F, Match

Short Answer

Average: 65%

Average: 88%

>00-

10

10_1

9

20-2

9

30-3

9

40-4

9

50-5

9

60-6

9

70-7

9

80-8

9

90-9

9

Pro

po

rtio

n o

f C

las

s (

%)

0

10

20

30

40

2015 TECHTEACH CONFERENCE - UIC

<gains> (ranges)

Learning Gains in Groups? 2016

<Gain> : Pretest (Individual) vs Post-test (Individual)Avg: 50%

>00-

10

10_1

9

20-2

9

30-3

9

40-4

9

50-5

9

60-6

9

70-7

9

80-8

9

90-9

9

Pro

po

rtio

n o

f C

las

s (

%)

0

10

20

30

40

2015 TECHTEACH CONFERENCE - UIC

<gains> (ranges)

Learning Gains in Groups? 2016

<Gain> : Post-test (Individual) vs Post-test (Individual)Avg: 66%

>00-

10

10_1

9

20-2

9

30-3

9

40-4

9

50-5

9

60-6

9

70-7

9

80-8

9

90-9

9

Pro

po

rtio

n o

f C

las

s (

%)

0

10

20

30

40

2015 TECHTEACH CONFERENCE - UIC

<gains> (ranges)

Learning Gains in Groups? 2016

<Gain> : Pretest (Individual) vs Post-test (Group)Avg: 80%

Final Grade

A+ A A- B+ B C+ C D+ D E F

Cla

ss

Pro

po

rtio

n (

%)

0

5

10

15

20

252016

Final Exam Scores

2015 TECHTEACH CONFERENCE - UIC

2015

2014

2013

20122011

Summary - Learning Outcomes(2011 versus 2012-2015)

2015 TECHTEACH CONFERENCE - UIC

Student performance

on exams

Learning gains pre/post assessments(Smith et al., 2008)

Final grades

7-10%

~10%

50-90%

2012-2015 – eInstruction, LectureTools and Echo360 Lecture Captures

enabled classrooms

Research Questions

2015 TECHTEACH CONFERENCE - UIC

1) Does the use of an Echo360-ALP feature influence the use of

another?

2) Is there a relationship between each individual metric and

exam outcomes?

3) Are there interactions between the metrics that lead to

differences in student performance and learning?

4) Is there a relationship between user profiles and exam

outcomes?

2015 TECHTEACH CONFERENCE - UIC

Does the use of an Echo360-ALP feature

influence the use of another?

R2 = 0.5449 (p<0.05)

2015 TECHTEACH CONFERENCE - UIC

Model generated by “stepwise

backward linear regression”.

• 6 of 256 interactions

were eliminated from

the model.

• Interactions between

the metrics are many

and complex.

Fit of the Linear Regression Model

Composite Exam Score (%)

Pre

dic

ted

Co

mp

osit

e

Exam

Sco

re (

%)

Are there interactions between the metrics that lead

to differences in exam outcomes?

2015 TECHTEACH CONFERENCE - UIC

40%

45%

50%

55%

Low Usage High Usage

Pre

dic

ted

learn

ing

gain

sLinear regression model

was used to evaluate the

effect size of each metric

on learning gains.

Effect size of each metric

calculated by setting all

metrics at their median

values except the one

under study.

Presentation engagement

video views

Activity score

Video engagement

Attendance

Activity participation

Note taking

Presentation views

Low usage (1st quartile) High usage (3rd quartile)

All metrics at median values

(2nd quartile)

What are the effects of the metrics

on the learning gains?

Using the Linear Regression Model to Evaluate the Effect Size

of Each Metric on Exam Outcomes

2015 TECHTEACH CONFERENCE - UIC

Is there a relationship between user profiles and

exam outcomes?

2015 TECHTEACH CONFERENCE - UIC

Example

Engagement score =

Weighting schemes which give the smallest interquartile range of engagement scores within each grade level

F

APP ATT NOT PVE VVE

0 1 0 0 0

D

APP ATT NOT PVE VVE

10 3 0 2 4

C

APP ATT NOT PVE VVE

8 0 0 0 1

B

APP ATT NOT PVE VVE

9 2 1 0 1

A

APP ATT NOT PVE VVE

9 2 0 2 0

2015 TECHTEACH CONFERENCE - UIC

2015 TECHTEACH CONFERENCE - UIC

What does it all mean to teaching and learning?

Acknowledgements

Teaching and Learning Support Services

Funding sources:

TLSS Pedagogical Grants

Vice-Rector Academics – Initiative Grants

Echo360 grant

uOttawa Blended Initiative

Ontario Consortium of Undergraduate Biology Educators

2015 TECHTEACH CONFERENCE - UIC