the achievement gap in online courses through a learning analytics lens
DESCRIPTION
Presentation at San Diego State University on April 12, 2013. Educational researchers have found that students from under-represented minority families and other disadvantaged demographic backgrounds have lower achievement in online (or hybrid) courses compared to face-to-face course sections (Slate, Manuel, & Brinson Jr, 2002; Xu & Jaggars, 2013). However, these studies assume that "online course" is a homogeneous entity, and that student participation is uniform. The content and activity of the course is an opaque "black box", which leads to conclusions that are speculative at best and quite possibly further marginalize the very populations they intend to advocate for. The emerging field of Learning Analytics promises to break open this black box understand how students use online course materials and the relationship between this use and student achievement. In this presentation, we will explore the countours of Learning Analytics, look at current applications of analytics, and discuss research applying a Learning Analytics research method to students from at-risk backgrounds. The findings of this research challenge stereotypes of these students as technologically unsophisticated and identify concrete learning activities that can support their success.TRANSCRIPT
John Whitmer, Ed.D.Academic Technology Services
California State University, Office of the Chancellor
San Diego State UniversityApril 12, 2013
The Achievement Gap in Online Courses through a Learning Analytics Lens
Motivating Questions …
1. Do our current uses of academic technologies (such as online learning) decrease or exacerbate the achievement gap?
2. If we don’t believe our current uses serve these students, how do we know? What can we do about it?
Agenda
1. Context: CSU Achievement Gap & Conceptual Framework
2. Recent Conventional Research in Online Courses
3. Research using Learning Analytics & Course Redesign
4. Next Steps & Discussion
1. CONTEXT: ACHIEVEMENT GAP & CONCEPTUAL FRAMEWORK
Increasing Access to Higher Education in the U.S.
Table adapted from data in NCES Digest of Educational Statistics (2011)
1976Percent of Enrollment
(1976)2010
Percent of Enrollment
(2010)
Percent Increase
All Students 10,986 21,016 91%White 9,076 83% 12,723 61% 40%Asian/Pacific Islander 198 2% 1,282 6% 548%
URM Students 1,493 14% 5,977 28% 300%American Indian 76 1% 196 1% 158%
Black 1,033 9% 3,039 14% 194%Hispanic 384 3% 2,741 13% 614%
Increased Access to Higher Education
All Students White Asian/Pacific Islander URM Students American Indian Black Hispanic0%
100%
200%
300%
400%
500%
600%
700%
Enrollment Increase by Race/Ethnicity in Higher Education (1976-2010)
Table adapted from data in NCES Digest of Educational Statistics (2011)
CSU Achievement Gap
By 2015, the CSU will improve graduation rates by 8 percentage points systemwide and halve the achievement gap. – Baseline 6-Year Graduation Rate: 46%– Target 6-Year Graduation Rate: 54%
– Baseline Achievement Gap: 11%– Target Achievement Gap: 5.5%
2
No Significant Difference: Framing Academic Technology
[academic technologies] are “mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition”. (Clark, 1983)
Image courtesy bsabarnowl @ Flickr
Index of studies: http://www.nosignificantdifference.org/
2. RECENT RESEARCH W/HYBRID & ONLINE COURSES
Adaptability to Online Learning: Differences Across Types of Students and Academic Subject Areas (2013: Xu, Jaggers)
Compares persistence and grade between online and f2f courses
Compares same student Washington Community and
Technical Colleges (2 yr) Studied 500,000 course
enrollments (10% online), 40,000 individuals
Data 2004-2009
Overall Findings
Table adapted from Xiu & Jaggers, 2013
Major Finding by Population
Overall Course ResultOnline GPA Average 2.77F2F GPA Average 2.98Entire Population -0.215Effect by Subject -0.267
Select Populations ResultBlack -0.394Males -0.288Academic Preparedness (F2F GPA<3.0 First Term) -0.314Age < 25 -0.300Cohort Effect (Courses w/+75% online at-risk students v. less than 25%) -0.359
Table adapted from Xiu & Jaggers, 2013
Major Finding by Subject
Overall Course ResultOnline GPA Average 2.77F2F GPA Average 2.98Entire Population -0.215Effect by Subject -0.267
Table adapted from Xiu & Jaggers, 2013
Select Subjects ResultsEnglish -0.394Applied Knowledge -0.322Social Science -0.308
Online Course
F2F CourseStudent
Grade
Grade
Treatments
Traditional Experimental Design
Image courtesy bsabarnowl @ Flickr
What’s your experience with Online or Hybrid Course Design?
Who has designed a fully online or hybrid course? (raise hands)
Of those who have, how many think it’s harder to create online/hybrid materials than to create face to face activities? (keep hands raised)
Peering into the blue box
Online CourseCours
e Desig
n
Faculty Course Development
Support (Reassigned
time, incentives, etc.) Student
use of online
materials
Specialist Support
(Instructional designers,
etc.)
Faculty &
Student
Training
Do we need *students* to adapt …
or
Do we need to change how *we* approach creating & evaluating
our technology-enhanced instructional materials?
3. LEARNING ANALYTICS & COURSE REDESIGN
Learner Analytics
“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)
Case Study: Intro to Religious Studies
• Undergraduate, introductory, high demand
• Redesigned to hybrid delivery format through “academy eLearning program”
• Enrollment: 373 students (54% increase on largest section)
• Highest LMS (Vista) usage entire campus Fall 2010 (>250k hits)
• Bimodal outcomes:• 10% increase on final exam• 7% & 11% increase in DWF
• Why? Can’t tell with aggregated data
54 F’s
LMS Use Variables
Administrative Activities (calendar, announcements)
Assessment Activities (quiz, homework, assignments, grade center)
Content Activities (web hits, PDF, content pages)
Engagement Activities (discussion, mail)
Student Characteristic Variables Enrollment Status First in Family to Attend
College Gender HS GPA Major-College Pell Eligible URM and Pell-Eligibility
Interaction Under-Represented
Minority URM and Gender
Interaction
Correlation: Student Char. w/Final Grade
Scatterplot of HS GPA vs. Course
Grade
Predict the trend
LMS use and final grade is _______ compared to student characteristics and final grade:
a) 50% smaller
b) 25% smaller
c) the same
d) 200% larger
e) 400% larger
Predict the trend
LMS use and final grade is _______ compared to student characteristics and final grade:
a) 50% smaller
b) 25% smaller
c) the same
d) 200% larger
e) 400% larger
Correlation LMS Use w/Final Grade
Scatterplot of Assessment Activity
Hits vs. Course Grade
Combined Variables Regression Final Grade by LMS Use & Student Characteristic Variables
LMS Use
Variables
25% (r2=0.25)
Explanation of change in final grade
Student Characteristic
Variables
+10%(r2=0.35)
Explanation of change in final grade
>
Correlation LMS & Student Characteristic Variables w/Final Grade
At-Risk Students: “Over-Working Gap”
Activities by Pell and Gradegrade / pelleligible
A B+ C C-
Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible
0K
5K
10K
15K
20K
25K
30K
35K
Value
Content
Content
Engage
Engage
Assess
Assess
Admin
Admin
Content
Content
Engage
Engage
Assess
Assess
Admin
Content
Content
Engage
Engage
Assess
Assess
Content
Content Engage
Engage
Assess
Assess
Admin
Admin
Measure Names
Admin
Assess
Engage
Content
Extra effort in content-related activities
Next Generation Learning Analytics
Graphic Courtesy Sasha Dietrichson, X-Ray Research SRL
COURSE REDESIGN Flagship: Program in Course Redesign, led by Carol Twig
(1999-2004)– Pew funded 30 grants, $8.8M budget to redesign courses
for improved outcomes & lower costs (institutionalization)
Result: 25 of 30 courses reported increased learning outcomes, 5 no change (not worse!)
– 17 reduction DWF (10-20%)– Cost reduction 20-77%, $3M annual savings
Adopted (with modifications by CSU, SDSU, Chico State)– Chico State evaluations:
http://www.csuchico.edu/academy/outcomes/index.shtml
Sample Improvements DWF (drop-failure-withdrawal) rates at Drexel were consistently reduced 10-12
percent in the redesigned course.
At OSU, withdrawals were reduced by 3 percent, failures by 4 percent and incompletes by 1 percent. As a result, 248 more students successfully completed the course compared to the traditional course.
At TCC, students in redesigned sections had a 68.4 percent success rate compared to 60.7 percent for traditional sections. Success rates were higher for all groups of students regardless of ethnicity, gender, disability, or original placement. The overall success rate for all composition students was 62 percent for the 2002-2003 year compared to 56 percent for the 1999-2000 year prior to redesign.
In the traditional course at USM, faculty-taught sections typically retained about 75 percent of students while adjunct- and TA-taught sections retained 85 percent. In the redesign, the retention rate was 87 percent. The rate of D and F grades dropped from 37 percent in the traditional course to 27 percent in the redesigned course. DFW rates dropped from 26 percent in the traditional course to 22 percent in the redesign.
Source: Program in Course Redesign Round III: Lessons Learned (http://www.thencat.org/PCR/R3Lessons.html)
Underserved Student Experiences More comfort, higher
participation in online forums
Appreciate ability to anonymously rewind / repeat / review materials
English language learners decreased social anxiety
Source: Twigg, C. (2005). Increasing Success for Underserved Students: Redesigning Introductory Courses. URL: http://www.thencat.org/Monographs/IncSuccess.htm
Proven PCR Techniques for Underserved Students
Interactive online tutorials (not PPT decks) Continuous assessment / feedback Increased student interaction Individualized, on-demand support Undergraduate learning assistants Structural supports that encourage student
engagement / progress
4. DISCUSSION
Discussion
Do you think there is an achievement gap at SDSU for under-served students in online / hybrid / tech enhanced courses? What evidence do you have for those beliefs?
What is SDSU doing about any existing achievement gap w/academic technology?
What supports are in place or could be developed?
Feedback? Questions?
John Whitmer [email protected]: johncwhitmer
Learning Analytics resources: http://www.johnwhitmer.net