does echo360‐alp analytics predict the relationship ... · • just -in time teaching f2f-class...

#THETA2017

This work is licensed under a Creative Commons Attribution 4.0 International License

Does Echo360‐ALP analytics predict the relationship between

student behaviours and learning?

Colin Montpetit, Sonya Sabourin

Leveraging the Echo360-ALP to Transform A Large Enrollment

Lecture Classroom into an Active Learning Zone

Colin Montpetit, Ph.D., Assistant Professor

Sonya Sabourin, B.Sc. Honour’s student

Department of Biology, University of Ottawa

2015 TECHTEACH CONFERENCE - UIC

Does Echo360-ALP analytics predict the relationship

between student behaviours and learning?


Modules (self-paced)• Targeted readings

• Interactive act./Videos

• Reflective act.

• Module quiz OnlineBlackBoard LMSInitial exposure

Summative• Two-Stage Collaborative Exams

(2 midterms + 1 final)

Assessments

Labs• Experiments

• Worksheets, reports

• Quizzes

F2F-Labs

Defined Learning Outcomes, 5E Learning Cycle

Class• Clicker case studies

• Peer-Instruction

• Just-in-Time Teaching

F2F-class

Echo360-ALP

65% 20%

15%

Echo360- Active Learning Platform



What is the class usage profile?


Engagement data provides insight via dashboards

Confusion

Questions

posed

Activity

answers/

participation

Notes

Attendance

Presentation

views

Learning Analytics Database

Lecture capture


Attendance, Participation, Activity Score


Note Taking


Presentation Views


Video Views

Caroline Petit-Turcotte Winter 2015

447 different students asked at least 1 question


Questions to the professorBIO1140 – Introduction to Cell Biology

153 different students flagged a slideSOS call out! (flagging slides)BIO1140 – Introduction to Cell Biology



Item Analysis – Difficulty Index


Item Analysis – Discrimination Index

0-10 10_19 20-29 30-39 40-49 50-59 60-69 70-79 80-89 90-99

Pro

po

rtio

n o

f C

las

s (

%)

0

5

10

15

20

Distribution of Learning Gains


<gains> (ranges)

2011: 30%

2014/16: 60%

Normalized LG = Post – Pre / 100 - pre

What Are Two-Stage Exam?


Stage 1: IndividualMCQs, T/F, Match

Short Answer

Stage 2: GroupsMCQs, T/F, Match

Short Answer

Average: 65%

Average: 88%

>00-

10

10_1

9

20-2

9

30-3

9

40-4

9

50-5

9

60-6

9

70-7

9

80-8

9

90-9

9

Pro

po

rtio

n o

f C

las

s (

%)

0

10

20

30

40


<gains> (ranges)

Learning Gains in Groups? 2016

<Gain> : Pretest (Individual) vs Post-test (Individual)Avg: 50%

>00-

10

10_1

9

20-2

9

30-3

9

40-4

9

50-5

9

60-6

9

70-7

9

80-8

9

90-9

9

Pro

po

rtio

n o

f C

las

s (

%)

0

10

20

30

40


<gains> (ranges)


<Gain> : Post-test (Individual) vs Post-test (Individual)Avg: 66%

>00-

10

10_1

9

20-2

9

30-3

9

40-4

9

50-5

9

60-6

9

70-7

9

80-8

9

90-9

9

Pro

po

rtio

n o

f C

las

s (

%)

0

10

20

30

40


<gains> (ranges)


<Gain> : Pretest (Individual) vs Post-test (Group)Avg: 80%

Final Grade

A+ A A- B+ B C+ C D+ D E F

Cla

ss

Pro

po

rtio

n (

%)

0

5

10

15

20

252016

Final Exam Scores


2015

2014

2013

20122011

Summary - Learning Outcomes(2011 versus 2012-2015)


Student performance

on exams

Learning gains pre/post assessments(Smith et al., 2008)

Final grades

7-10%

~10%

50-90%

2012-2015 – eInstruction, LectureTools and Echo360 Lecture Captures

enabled classrooms

Research Questions


1) Does the use of an Echo360-ALP feature influence the use of

another?

2) Is there a relationship between each individual metric and

exam outcomes?

3) Are there interactions between the metrics that lead to

differences in student performance and learning?

4) Is there a relationship between user profiles and exam

outcomes?


Does the use of an Echo360-ALP feature

influence the use of another?

R2 = 0.5449 (p<0.05)


Model generated by “stepwise

backward linear regression”.

• 6 of 256 interactions

were eliminated from

the model.

• Interactions between

the metrics are many

and complex.

Fit of the Linear Regression Model

Composite Exam Score (%)

Pre

dic

ted

Co

mp

osit

e

Exam

Sco

re (

%)

Are there interactions between the metrics that lead

to differences in exam outcomes?


40%

45%

50%

55%

Low Usage High Usage

Pre

dic

ted

learn

ing

gain

sLinear regression model

was used to evaluate the

effect size of each metric

on learning gains.

Effect size of each metric

calculated by setting all

metrics at their median

values except the one

under study.

Presentation engagement

video views

Activity score

Video engagement

Attendance

Activity participation

Note taking

Presentation views

Low usage (1st quartile) High usage (3rd quartile)

All metrics at median values

(2nd quartile)

What are the effects of the metrics

on the learning gains?

Using the Linear Regression Model to Evaluate the Effect Size

of Each Metric on Exam Outcomes


Is there a relationship between user profiles and

exam outcomes?


Example

Engagement score =

Weighting schemes which give the smallest interquartile range of engagement scores within each grade level

F

APP ATT NOT PVE VVE

0 1 0 0 0

D

APP ATT NOT PVE VVE

10 3 0 2 4

C

APP ATT NOT PVE VVE

8 0 0 0 1

B

APP ATT NOT PVE VVE

9 2 1 0 1

A

APP ATT NOT PVE VVE

9 2 0 2 0



What does it all mean to teaching and learning?

Acknowledgements

Teaching and Learning Support Services

Funding sources:

TLSS Pedagogical Grants

Vice-Rector Academics – Initiative Grants

Echo360 grant

uOttawa Blended Initiative

Ontario Consortium of Undergraduate Biology Educators


does echo360‐alp analytics predict the relationship ... · • just -in time teaching f2f-class...

Documents