Why Predictive Modeling?Predict future events using data already

available.

E-Learning schools collect lots of behavioral data useful in predictive modeling.Detailed tracking of student activities.

Logins Submissions Class discussions etc.

Data SourcesEnrollment histories and demographics

Online student activity

Advisement records

Assignment & test scores

GoalsConstruct classification models to identify online

students at-risk of not successfully completing their course.Course Success = Student completes course w/ ‘C’ or

better.

Estimate the probability that a student will be unsuccessful.Separate High, Medium, and Low risk students.

Perform predictions immediately after the 1st week of class.

VariablesEnrollment History

New student?Has previously been successful in any course?Has previously been unsuccessful in any

course?Is student re-taking this course?Has taken a Developmental level course?etc.

Variables (Cont’d)Current Enrollment

Taking other classes as of start date?Taking more than 6 other credits as of start date?etc.

Online ActivityLogged in before start date?Logged in on 1st day?Logged in on 2nd day?Logged in during 1st week?Has opened an assessment during 1st week?etc.

Variables (Cont’d)Financials

On financial aid?

AdvisingHad previously scheduled an advising

appointment?

DemographicsAgeGender

Naïve Bayesian ModelOur models are constructed using the Naïve

Bayesian technique.

Chosen for its accuracy and robustness.

Every variable always “has its say” on the prediction.Unlike other popular methods (i.e. decision trees).

Presents fair interpretation of student’s likelihood to succeed.

Naïve Bayesian Model (Cont’d)How does it work?

Model takes input from each variable independently.

Each variable has unique influence on result.

Weight of influence based on how often that variable has been associated with previous cases of success. Some variables more predictive – have more

influence.

Naïve Bayesian Model (Cont’d)Most significant variables

Final success probability derived from combined input of all variables.

Increases Probability of Success? Variable Description

Yes Logged in to course homepage during first week of class.Yes Logged in to course homepage on first day of class.Yes Logged in to course homepage prior to start date.No Student taking other classes at Rio Salado College simultaneously.Yes Student has been successful in a previous course at Rio Salado College.No Student has been unsuccessful in a previous course at Rio Salado College.No Student is re-taking the course.

Naïve Bayesian Model (Cont’d)Predicted outcome

Risk levelRisk Level Probability of Success

Green Greater than or equal to 70%

Yellow Between 30% and 70%

Red Less than or equal to 30%

Predicted Outcome

Probability of Success

Success Greater than 50%

Non-Success Less than or equal to 50%

DemonstrationSpring 2009 Students

Taken from Humanities course - Start date January 26th

Student Final Grade Probability of SuccessStudent 1 ? ?

Student 2 ? ?Student 3 ? ?Student 4 ? ?Student 5 ? ?Student 6 ? ?Student 7 ? ?

Student 8 ? ?Student 9 ? ?

Student 10 ? ?Student 11 ? ?Student 12 ? ?Student 13 ? ?Student 14 ? ?

Student 15 ? ?Student 16 ? ?Student 17 ? ?Student 18 ? ?

DemonstrationSpring 2009 Students

Taken from Humanities course - Start date January 26th

Student Final Grade Probability of SuccessStudent 1 ? 100%

Student 2 ? 94%Student 3 ? 88%Student 4 ? 82%Student 5 ? 76%Student 6 ? 70%Student 7 ? 64%

Student 8 ? 64%Student 9 ? 58%

Student 10 ? 52%Student 11 ? 46%Student 12 ? 40%Student 13 ? 34%Student 14 ? 28%

Student 15 ? 22%Student 16 ? 22%Student 17 ? 22%Student 18 ? 16%

DemonstrationSpring 2009 Students

Taken from Humanities course - Start date January 26th

Student Final Grade Probability of SuccessStudent 1 ? 100%

Student 2 ? 94%Student 3 ? 88%Student 4 ? 82%Student 5 ? 76%Student 6 ? 70%Student 7 ? 64%

Student 8 ? 64%Student 9 ? 58%

Student 10 ? 52%Student 11 ? 46%Student 12 ? 40%Student 13 ? 34%Student 14 ? 28%

Student 15 ? 22%Student 16 ? 22%Student 17 ? 22%Student 18 ? 16%

DemonstrationSpring 2009 Students

Taken from Humanities course - Start date January 26th

Student Final Grade Probability of SuccessStudent 1 100%

100%Successful

Student 2 94%Student 3 88%Student 4 82%Student 5 76%Student 6 70%Student 7 64%

43%Successful

Student 8 64%Student 9 58%

Student 10 52%Student 11 46%Student 12 40%Student 13 34%Student 14 28%

20%Successful

Student 15 22%Student 16 22%Student 17 22%Student 18 16%

Success rates calculated after final grades recorded

Model ValidationModel applied to Fall 08 and Spring 09

enrollments from select disciplines1.Predictions compared to outcomes already

recorded in student information system.

How accurately does model predict correct outcome?Predicted Outcome

Successful Unsuccessful

59% 41% SuccessfulActual Outcome

30% 70% Unsuccessful

1: Includes select courses from Science, Biology, English, Math, Languages, Communication, Social Sciences, Humanities/History, and Reading. Students in special programs, such as dual enrollment, military, and incarcerated re-entry were not included. Run Aug 09.

Model Validation (Cont’d)How well does model assign students to risk

levels?

“Yellow” StudentsGreen students succeed most often, Red

students succeed least often, and Yellow students fall somewhere in the middle.

Yellow students do not show a strong tendency towards either outcome – could go either way.

Moral obligation to help Yellow students succeed.More on Red students a bit later…

Retaining the online student“Adult learning theories are built on the

premise that teachers will assist their students to become self-directed and independent” (Muirhead and Min, 2001, p. 1).

How does this best work online?

Retaining the online studentResearch suggests that students are unaware

of what strategies are needed to be self-monitoring online learners (Muirhead and Min, 2001; Ormrod, 2004; Youngblood, 2001) .

This is the challenge and area faculty need to focus on to increase student retention and success.

Share your thoughts…Take a moment, what strategies can you

incorporate online to help your students become self-monitoring learners?

How can we help our students become effective online learners?

Focusing interventionsCourse design and interactions with

instructors have been identified as key areas for online student success (Hillsheim, 1998; Paloff & Pratt, 2003; Swan, 2001)

The Rio Model is one course many instructors, so the majority of our instructors do not control the course design.

What we do before the interventions…To enable students to be self-monitoring, our

courses are designed to engage students in self-check activities

Students are taught to monitor their progress by completing pre and post tests

Students are engaged in summarizing lessonsBest practices in andragogy are embedded in

our online courses…

We try to go beyond this with additional Faculty Interventions…

Department InterventionsCommunication Department:

Students receive a phone call from the instructor during 2nd week, plus follow-up phone call 1 week prior to midpoint.

Humanities/History Department: Phone call from instructional helpdesk.

Social Sciences Department: Students receive a phone call from instructor

Phone Call to Student“I am calling because you are currently in your

2nd week of your course. I want to make sure you have a successful experience. The first thing you can do to ensure this occurs is to be sure to communicate your content questions to me. If you do not understand something you have read or are attempting to answer, be sure to ask. If you are still struggling to grasp the content, you may want to speak with a tutor. We offer free tutoring at Rio Salado college, both in-person and online. Do you currently have any questions that I can assist you with?”

Student feedback…Student appreciate the time and effort put forth

by the instructor!

“Eight weeks ago I came into this course very unsure if I could do this extremely intense course, but I have made it, with your help and guidance I have learned so much in such a short period of time. I now find myself looking at the world a lot differently.” –Rio Student

Student feedback“Now that the course is over and grades are

posted I just wanted to thank you for your help with this class.  I have to tell you that this class was one of the classes I was dreading most on my prereq list, because I remember being so lost with it in high school.  Your quick feedback (and patience) with my numerous questions was a lifesaver”.

“It is really hard to get to know your instructors in an online class since you don't actually meet, but I wanted to let you know you are by far the friendliest, most helpful, upbeat, and prompt professor I have this semester, and I am taking 18 credits, so thats saying something”. 

Student Success Interventions3 disciplines conducted initial intervention

trials in Summer I 2009 using the models described previously.

Focus on Yellow students.

Random 50% receive intervention, other 50% placed in control group for future comparison.1

Intervention strategies varied by discipline.Designed by faculty chair.

1: Control group students still had access to all Rio Salado College services and were still exposed to the traditional forms of success and retention outreach efforts that all students receive.

Preliminary ResultsCategory

Successful EnrollmentsNN %

Green 165 86.8% 190Yellow (Intervention Group) 86 75.4% 114Yellow (Control Group) 71 69.6% 102Red 45 54.9% 82Overall 367 75.2% 488

*Preliminary results only include Summer I students graded out as of 9/23/09.

Preliminary Results (Cont’d)Intervention success rate 8% higher than

control.

Not statistically significant at the 0.05 or 0.10 levels.

Results are preliminary – sample size currently too small to make strategic decisions.More on that a bit later…

Preliminary Results (Cont’d)Online activity is a significant factor in course

success.One of the potential metrics of course

engagement.

How did online activity in intervention group compare to control group?Category

Average Log-In Total

Average Weekly Log-In Rate N

Green 81.8 7.8 190Yellow (Intervention Group) 78.7 6.8 114Yellow (Control Group) 67.4 6.1 102Red 53.8 4.5 82Overall 73.3 6.7 488

Preliminary Results (Cont’d)Average log-in total for intervention group

17% higher than control.Difference is statistically significant at the 0.05

level.1

Average weekly log-in rate also higher, but only at the less significant 0.10 level.2

Interventions applied to Yellow students may have influenced their online behavior.

1: One-tailed t-test; p-value = 0.047. 2: One-tailed t-test; p-value = 0.085.

Future WorkOngoing intervention trials in Summer II and

Fall semesters.

Continue improving interventions based on results of controlled trials.

Collect sample size large enough to disaggregate results and determine which strategies worked best.

Future Work (Cont’d)College also researching new models capable

of generating prediction at time of registration.Front-line staff can intervene with Yellow and

Red students as early as possible.Will create two pronged approach when

combined with faculty-designed interventions.Phase II – FY10-11 Phase I – FY09-10

Tell us where you are…

Has anyone used predictive modeling? Plans to include it?

What have your results been?

Questions?Please feel free to contact us:

Shannon Corona, Physical Science Faculty Chair

Shannon.corona@riosalado.edu or 480-517-8285

Adam Lange, Program AnalystAdam.lange@riosalado.edu or 480-517-8401