using an online course to support instruction of introductory statistics

Using an Online Course to Using an Online Course to Support Instruction of Support Instruction of Introductory StatisticsIntroductory Statistics

CAUSE Webinar CAUSE Webinar (8/14/2007)(8/14/2007)

Oded MeyerOded Meyer

Dept. of StatisticsDept. of StatisticsCarnegie Mellon UniversityCarnegie Mellon University

IntroductionIntroduction Educational Mission of FunderEducational Mission of Funder (The William and Flora Hewlett Foundation)(The William and Flora Hewlett Foundation)

Provide open access to high quality post-secondary Provide open access to high quality post-secondary education and educational materials to those who education and educational materials to those who otherwise would be excluded due to:otherwise would be excluded due to: Geographical constraintsGeographical constraints Financial difficultiesFinancial difficulties Social barriersSocial barriers

To meet this goal:To meet this goal: A complete stand-alone web-based introductory A complete stand-alone web-based introductory

statistics course.statistics course. openly and freely available to individual learners openly and freely available to individual learners

online.online.

Moving Instruction Out of the Moving Instruction Out of the Classroom: ChallengesClassroom: Challenges

Course Organization and StructureCourse Organization and Structure Students often view what they learn as a set of Students often view what they learn as a set of

isolated facts. isolated facts.

Instructor promotes coherence, sets course path.Instructor promotes coherence, sets course path.

Online course: high level of scaffolding in Online course: high level of scaffolding in structure is needed. structure is needed.

Course is organized around the Big Picture. Course is organized around the Big Picture. Rigid structure throughout material hierarchy.Rigid structure throughout material hierarchy. Smooth conceptual path.Smooth conceptual path.

Challenges (cont.)Challenges (cont.) Effective Use of Media ElementsEffective Use of Media Elements

Course follows well researched principles to Course follows well researched principles to minimize cognitive load imposed by the minimize cognitive load imposed by the learning design. For example…learning design. For example…

Best to reinforce information over auditory Best to reinforce information over auditory and visual channels simultaneously.and visual channels simultaneously.

Challenges (cont.)Challenges (cont.) Immediate and Targeted FeedbackImmediate and Targeted Feedback

Studies: immediate feedback Studies: immediate feedback students students achieve desired level of performance faster.achieve desired level of performance faster.

We needed to compensate for no immediate We needed to compensate for no immediate instructor – students feedback loops.instructor – students feedback loops.

Throughout the course immediate and Throughout the course immediate and tailored feedback is given.tailored feedback is given.

mini tutors embedded in the material.mini tutors embedded in the material. self assessments activities (Did I get this?)self assessments activities (Did I get this?)

Course EvaluationCourse Evaluation

““Do No Harm” Study (Fall 2005):Do No Harm” Study (Fall 2005): Online course vs. traditional course at CMU.Online course vs. traditional course at CMU.

Traditional Intro. Stats. Course:Traditional Intro. Stats. Course: Three 50 min. lectures a week.Three 50 min. lectures a week. One lab a week (approx. 1 TA per 10 students).One lab a week (approx. 1 TA per 10 students). Weekly HW assignments.Weekly HW assignments. Text: Intro Practice Stats (Moore & McCabe, 2006).Text: Intro Practice Stats (Moore & McCabe, 2006). Evaluation: three midterms + comprehensive final.Evaluation: three midterms + comprehensive final.

Evaluation: First Study (cont.)Evaluation: First Study (cont.)

Sample (online section)Sample (online section):: Students were invited to participate in “online Students were invited to participate in “online

section”.section”. Of those who volunteered, 20 students were Of those who volunteered, 20 students were

chosen randomly and reasonably resembled the chosen randomly and reasonably resembled the entire class in terms of gender, race and prior entire class in terms of gender, race and prior exposure to statistics.exposure to statistics.

RequirementsRequirements:: go through the course in a specified pace and go through the course in a specified pace and

complete all activities.complete all activities. attend a weekly 50 min. meeting for feedback attend a weekly 50 min. meeting for feedback

about their learning experience & questions.about their learning experience & questions. Evaluation:Evaluation: three midterms + comprehensive three midterms + comprehensive

final (matched in level of difficulty to rest of the final (matched in level of difficulty to rest of the class).class).

Evaluation: First Study (cont.)Evaluation: First Study (cont.)

Results:Results: All but 2 students followed schedule (with up to All but 2 students followed schedule (with up to

two days of delay).two days of delay).

Three instances of clarifications (regression line, Three instances of clarifications (regression line, sampling distributions, p-value).sampling distributions, p-value).

PerformancePerformance::

Evaluation: First Study Evaluation: First Study (cont.)(cont.)

ExamExam Numerical Numerical MeasuresMeasures

TraditionTraditionalal

OnlineOnline

First ExamFirst ExamEDA , EDA , Sampling, Sampling, DesignDesign

Sample SizeSample Size

MeanMean

Stand. Dev.Stand. Dev.

201201

90.1790.17

8.598.59

202088.7588.75

6.236.23

Second Second ExamExamProbabilityProbability


MeanMean


202202

81.8281.82

14.2514.25

2020

81.4581.45

13.8213.82

Third ExamThird ExamInferenceInference


MeanMean


201201

85.8785.87

11.9111.91

2020

85.1085.10

16.8016.80

FinalFinalComprehensivComprehensivee


MeanMean


204204

83.5483.54

11.0611.06

2020

84.7984.79

12.2312.23

Performance:Performance:

Evaluation: (cont.)Evaluation: (cont.)

Second Study (Spring 2006):Second Study (Spring 2006): Measuring statistical literacy - CAOS Test. Measuring statistical literacy - CAOS Test. CComprehensive omprehensive AAssessment of ssessment of OOutcomes utcomes

in a first in a first SStatistics course)tatistics course)(delMas, Ooms, Garfield, Chance)(delMas, Ooms, Garfield, Chance)

40 multiple choice items40 multiple choice items Measures statistical literacy & conceptual Measures statistical literacy & conceptual

understanding.understanding. Focus on reasoning about variability.Focus on reasoning about variability. 18 expert raters agreed with the statement:18 expert raters agreed with the statement:

““CAOS measures outcomes for which I would be CAOS measures outcomes for which I would be disappointed if they were not achieved by disappointed if they were not achieved by

students who succeed in my statistics courses.”students who succeed in my statistics courses.”

Evaluation: Second Study Evaluation: Second Study (cont.)(cont.)

CMU SampleCMU Sample:: 27 students, same selection process as in first study.27 students, same selection process as in first study. Same course structure and requirements as in first study.Same course structure and requirements as in first study. Students took the CAOS test as a pretest (n=27), and Students took the CAOS test as a pretest (n=27), and

then as a posttest (n=24). then as a posttest (n=24).

National CAOS Sample: National CAOS Sample: (delMas et al., AERA 2006)(delMas et al., AERA 2006) 488 students, 18 instructors, 16 institutions, 14 states.488 students, 18 instructors, 16 institutions, 14 states. 2 yr./tech.: 12.5%, 4 yr. college: 41.6%, Univ.: 45.9%2 yr./tech.: 12.5%, 4 yr. college: 41.6%, Univ.: 45.9% Prerequisite: no math (28.9%), HS algebra (46.1%) ,Prerequisite: no math (28.9%), HS algebra (46.1%) , college algebra (20.7%), calculus (4.3%)college algebra (20.7%), calculus (4.3%)


Three instances of clarifications (correlation, Three instances of clarifications (correlation, binomial distribution, sampling distributions).binomial distribution, sampling distributions).

nn Average % Average % correctcorrect

PretestPretest 488488 43.343.3

PosttePosttestst

488488 51.251.2

Increase: 7.9% Increase: 7.9% [t(487) = 13.8, p <.001][t(487) = 13.8, p <.001]


PretesPretestt

2424 55.855.8

PosttePosttestst

2424 66.566.5

Increase: 11.7%Increase: 11.7%[t(23) = 4.7, p <.001][t(23) = 4.7, p <.001]

Results:Results:

National CAOS Sample:National CAOS Sample:

CMU Sample:CMU Sample:

Results (cont.)Results (cont.) Measured outcomeMeasured outcome†† of items with less than of items with less than

50% of students correct on posttest:50% of students correct on posttest: Understanding of the purpose of randomization in an Understanding of the purpose of randomization in an

experiment (29.2%).experiment (29.2%).

MisconceptionsMisconceptions: reduces sampling error, increase : reduces sampling error, increase accuracy of results.accuracy of results.

Understand how sampling error is used to make an Understand how sampling error is used to make an informal inference about a sample mean (8.3%).informal inference about a sample mean (8.3%).

Common mistakeCommon mistake (62.5%): basing inference on the (62.5%): basing inference on the sample SD, disregarding the sample size.sample SD, disregarding the sample size.

† † as defined in delMas et al. AERA, 2006as defined in delMas et al. AERA, 2006



Understand how sampling error is used to make an Understand how sampling error is used to make an informal inference about a sample mean (8.3%).informal inference about a sample mean (8.3%).

Common mistakeCommon mistake (62.5%): basing inference on the (62.5%): basing inference on the sample SD, disregarding the sample size.sample SD, disregarding the sample size.

Understanding of the factors that allow generalizing Understanding of the factors that allow generalizing sample results to the population (45.8%).sample results to the population (45.8%).

MisconceptionMisconception: if the sample is small relative to the : if the sample is small relative to the population, generalizing results is problematicpopulation, generalizing results is problematic..

Understanding of the logic of significance test when Understanding of the logic of significance test when the null hypothesis is rejected (41.7%).the null hypothesis is rejected (41.7%).

MisconceptionMisconception: rejecting the null : rejecting the null null is false. null is false.

ResultsResults (cont.) (cont.) Items with < 50% of CAOS sample correctItems with < 50% of CAOS sample correct and ≥ 50% of CMU sample correct on posttestand ≥ 50% of CMU sample correct on posttest

Describing the distribution of a quantitative variable.Describing the distribution of a quantitative variable. Mean>Median Mean>Median the distribution is most likely the distribution is most likely

skewed left.skewed left. Interpretation of a boxplot.Interpretation of a boxplot. Correctly estimate and compare SD’s for different Correctly estimate and compare SD’s for different

histograms.histograms. Correlations does not imply causation.Correlations does not imply causation. Understanding that statistics from small samples vary Understanding that statistics from small samples vary

more than those from large samples.more than those from large samples. Understanding of expected patterns in sampling Understanding of expected patterns in sampling

variabilityvariability Selecting appropriate sampling distribution for Selecting appropriate sampling distribution for

particular population and sample sizeparticular population and sample size


Evaluation (cont.)Evaluation (cont.)

To summarize the results so far…To summarize the results so far… As far as performance and achieving As far as performance and achieving

statistical literacy… the online course statistical literacy… the online course definitely “does no harm” .definitely “does no harm” .

For some traditionally difficult statistical ideas For some traditionally difficult statistical ideas (in EDA and some aspects of understanding (in EDA and some aspects of understanding variability) the online course variability) the online course mightmight have a have a slight edge over traditional courses.slight edge over traditional courses.

Given that the course was administered Given that the course was administered almost “stand-alone”, this was quite almost “stand-alone”, this was quite encouraging.encouraging.

Evaluation: summary (cont.)Evaluation: summary (cont.)

The CAOS test pin-pointed important The CAOS test pin-pointed important statistical ideas that the online course did statistical ideas that the online course did not succeed in conveying, and revealed not succeed in conveying, and revealed which misconceptions need to be rooted which misconceptions need to be rooted out.out.

Students seem to find the course “friendly”.Students seem to find the course “friendly”. All students reported at least some increase in All students reported at least some increase in

their interest in statistics.their interest in statistics.

75% 75% Definitely RecommendDefinitely Recommend 25% Probably Recommend25% Probably Recommend 0% Probably not Recommend0% Probably not Recommend 0% Definitely not Recommend0% Definitely not Recommend

Student Quotes Student Quotes

““I really like the way you can learn individually I really like the way you can learn individually and at your own pace. If I understand and at your own pace. If I understand something, I can move through it quickly and something, I can move through it quickly and take more time on challenging things.”take more time on challenging things.”

"This is so much better than reading a "This is so much better than reading a textbook or listening to a lecture! My mind textbook or listening to a lecture! My mind didn’t wander, and I was not bored while didn’t wander, and I was not bored while doing the lessons. I actually doing the lessons. I actually learned learned something."something."

Evaluation (cont.)Evaluation (cont.)Third Study: Accelerated Third Study: Accelerated LearningLearning Study (Spring Study (Spring

07)07)

Accelerated online course vs. “traditional control”.Accelerated online course vs. “traditional control”. Students could choose to register for an accelerated “online Students could choose to register for an accelerated “online

section” (8 weeks instead of 15 weeks)section” (8 weeks instead of 15 weeks) 25 students were selected at random. 25 students were selected at random.

Those not chosen Those not chosen “traditional control “traditional control”.”.

Requirements:Requirements: Go through the course in an accelerated pace and Go through the course in an accelerated pace and

complete all the activities.complete all the activities. Post questions that they wanted addressed in class.Post questions that they wanted addressed in class.

Attend Attend twotwo 50 minute meetings a week for “focused 50 minute meetings a week for “focused lectures”, where we went through more examples that lectures”, where we went through more examples that targeted topics/issues that students were struggling with.targeted topics/issues that students were struggling with.

Evaluation: Third Study Evaluation: Third Study (cont.)(cont.) Results:Results:


PretestPretest 2121 5656

PosttePosttestst

2121 73.573.5


PretesPretestt

2424 55.855.8

PosttePosttestst

2424 66.566.5

Online accelerated courseOnline accelerated course::

Online course group (second study):Online course group (second study):

Increase: 11.7%Increase: 11.7%[t(23) = 4.7, p <.001][t(23) = 4.7, p <.001]


Evaluation: Third Study Evaluation: Third Study (cont.)(cont.)


PretestPretest 2121 5656

PosttePosttestst

2121 73.573.5


PretesPretestt

4040 5050

PosttePosttestst

4040 5353

Online accelerated courseOnline accelerated course::

Traditional control:Traditional control:

Increase: 3%Increase: 3%


OLI students showed significantly greater OLI students showed significantly greater gains (pre to post) than the Traditional gains (pre to post) than the Traditional “control” students on the CAOS test.“control” students on the CAOS test.

0

0.05

0.1

0.15

0.2

0.25

OLI TRAD

Gain

Score

17.5%17.5%

3%3%

These effects need to be considered in light of These effects need to be considered in light of the significant difference between groups at the significant difference between groups at pretest (even after our stratified randomized pretest (even after our stratified randomized assignment to groups).assignment to groups).

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

OLI TRD

Pre

test Pro

port

ion C

orr

ect

56%56%50%50%

Investigating the pretest scores further, there Investigating the pretest scores further, there is a significant linear relationship between is a significant linear relationship between pretest and posttest score.pretest and posttest score.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

Pretest CAOS Proportion

Postt

est CAO

S P

roport

ion

After accounting for the pretest’s After accounting for the pretest’s predictiveness, (ANCOVA) there is still a predictiveness, (ANCOVA) there is still a significant advantage for OLI students.significant advantage for OLI students.

Summary of third study & final Summary of third study & final thoughts:thoughts:

The online students gained much more (on the The online students gained much more (on the CAOS test) than did the “traditional controls”.CAOS test) than did the “traditional controls”.

This is noteworthy given that the OLI students This is noteworthy given that the OLI students had half a semester to cover a semester’s had half a semester to cover a semester’s worth of material.worth of material.

I believe that the gain in the third study I believe that the gain in the third study (course + focused lectures format) was better (course + focused lectures format) was better than the gain in the second study (stand-alone than the gain in the second study (stand-alone format) because the course was developed as format) because the course was developed as a stand-alone course (how ironic…) a stand-alone course (how ironic…)

An issue that needs to be examined is the An issue that needs to be examined is the effect of the accelerated learning on effect of the accelerated learning on retention (a follow-up study is planned in retention (a follow-up study is planned in down-stream courses).down-stream courses).

The format of the third study was among The format of the third study was among the best teaching experiences I’ve had in the best teaching experiences I’ve had in my 15 years of teaching statistics.my 15 years of teaching statistics.

I strongly believe (and hope, maybe…) that I strongly believe (and hope, maybe…) that no online course will ever be able to replace no online course will ever be able to replace an enthusiastic and engaging teacher. an enthusiastic and engaging teacher. However…However…

Having the students engage with material Having the students engage with material on their own using an online course on their own using an online course supplemented by focused lectures is a supplemented by focused lectures is a “winning combination”.“winning combination”.

Contact Information:Contact Information:

Oded MeyerOded Meyer

[email protected]@stat.cmu.edu

To access the course:To access the course:

go to: go to: www.cmu.edu/oli/www.cmu.edu/oli/ and follow the and follow the link to the statistics course.link to the statistics course.

mailto:[email protected]

http://www.cmu.edu/oli/

using an online course to support instruction of introductory statistics

Documents