Lessons from the Real World

Oct 2016 Deploying Open Learning Analytics at National Scale

Michael WebbDirector of Technology and Analytics, Jisc

Eitel J.M. Lauría, PhDProfessor of Information Technology & Systems, Marist College

Kate ValentiVice President of Operations, Unicon

Agenda

» Strategic View

› A brief introduction to Learning Analytics

› National issues in the UK

» Technical View

› Open architecture

› Predictive modeling

» Implementation View

› Trends and tactics from the field

» Discussion

A brief introduction to Learning Analytics

Our working definition...

What do we mean by Learning Analytics?

» The application of big data techniques such as machine based learning and data mining to help learners and institutions meet their goals:

› For our project:

– Improve retention (current project)

– Improve achievement (current project)

– Improve employability (current project)

– Improve learning design (later stage)

Learning Analytics stages get progressively “smarter”

Basic Analytics

What has happened

Automated Analytics

What is happening

Predictive Analytics

What might happen

A Strategic OverviewNational and institutional strategic issues

National issues in the UK: Retention

» 16-18 Education:

› 178,100 students aged 16-18 failed to finish (2012/13)

› costing UK £814 million a year

» Undergraduates:

› 8% of undergraduates drop out in their first year of study

› This costs universities up to £33,000 per student

National issues in the UK: Differential achievement

» Parental background and ethnicity impact achievement:

National issues in the UK: Differential achievement

2/03/2016 The case for Learning Analytics

» Which behaviours are associated with lower than expected academic achievement?

National issues in the UK: Teaching excellence framework

Technical Overview What does the architecture look like?

Jisc’s Learning Analytics project

Three core strands

Learning Analytics architecture and

serviceToolkit Community

Jisc Learning Analytics

Toolkit: Code of practice

2/03/2016 The case for Learning Analytics

Jisc Learning Analytics architecture

What

» Building a national architecture

» Defined standards and models

» Implementation with core services

Why?

» Standards mean models, visualisations and so on can be shared

» Lower cost per institutions through shared infrastructure

» Lower barrier to innovation – the underpinning work is already done

What do we mean by an open architecture?

» All APIs published, and process for engaging in their development

» Open standards and definitions

› Data Models and Definitions Creative Commons.

› Developed openly on Github

» All core elements open source or open specification (eg creative commons)

» Freedom to implement both commercial and open solutions as the non-core elements

Data Collection

DataStorageand Analysis

Presentation and Action

Jisc Learning Analytics open architecture: core

Alert and Intervention system

Staff Dashboards Consent Student App

LearningAnalytics Processor

LearningRecords Warehouse

Student Records VLE Library

DataExplorer

Self Declared Data

Meanwhile, in the US...Learning Analytics Processor: Predictive Modeling Framework

Motivation: Alarming Stats in 2010

36% 4-year completion rate across all four-year institutions in the US

21% for Black students

25% for Hispanic students

58% 6-year completion rate for four-year institutions

40% for Black students

49% for Hispanic Students

41% 25-to-34 Year-Olds with an Associate Degree or Higher (US ranked 12th among 36 developed nations)

Sources: U.S. Dept. of Education, Postsecondary Education Data System (2009) CollegeBoard, Advocacy & Policy Center, The Completion Agenda 2011 Progress Report

Open Academic Analytics Initiative @ Marist

EDUCAUSE Next Generation Learning Challenges (NGLC) grant

Funded by Bill and Melinda Gates Foundation

Use machine learning to find patterns in large datasets as means to predict student academic performance.

Create “early alert” framework:

• Predict academically at-risk students in initial weeks of a course

• Deploy intervention to improve chances of success

Based on Open ecosystem for academic analytics

• Sakai Collaboration and Learning Environment

• Pentaho Business Intelligence Suite (Kettle + Weka)

• Collaboration with commercial vendors (IBM SPSS Modeler)

Learning Analytics Processor @ Marist: Early Alert How does it actually work?

(binary classification problem)

Hardware Platform: IBM zEnterprise 114 with BladeCenter Extension (zBX) Virtualized Servers: 64 bit, 16/32 GB RAM Linux Red Hat

Extraction, Transformation &

Loading

Scoring(predictions on new student data using library of persisted learnt classifiers)

Predictive Model Building

(classifiers learnt from data)

New StudentData

(early in the Semester)

Prediction of at-risk studentsSingle node architecture

Relational Storage

Intervention

SATs, GPA,HS ranking, Course size,Course grade(target feature)

Age, gender,ethnicity,income level

SessionsResourcesLessonsAssignmentsForumsTests

Partialcontributionsto final grade

Logistic RegressionSVMsNaïve BayesJ48 Decision Trees

Student Academic

Data

Student Demographic

Data

LMS Event Log Data

LMS Gradebook Data

Learning Analytics Processor @ Marist: Early Alert New Iteration: Cluster Computing Architecture

New StudentData

(early in the Semester)

Prediction of At-risk students

Intervention

Scoring(predictions on

new student data using library of persisted learnt

classifiers)

Hardware Platform (Dev) Linux VMs (32GB RAM) running on IBM PureFlex System

Distributed Storage (HDFS)& Processing

Extraction, Transformation &

Loading

Predictive Model Building

(classifiers learnt from data)

Job

Sc

hed

ulin

g

Student Academic Data

Student Demographic Data

LMS Event Log Data

LMS Gradebook Data

Library Data

Student Engagement Data

Social Network Data

and more …

CURRENT

FUTURE

Scales well for Big Data use cases(more volume & variety)

Logistic RegressionRandom ForestsNaïve Bayes

Promising Outcomes

Phase II: Cluster Computing Accuracy Recall FP Rate

Marist

- 3 semesters, 25K records each 86% 87% 14%

North Carolina State University

- 3 semesters, 160K recs each 81% 77% 18%

- 3 semesters, online, 85K recs each 80% 82% 19%

Jisc Project:

• 260,000 records

• 4 institutions (Aberystwyth University, University of Gloucestershire, Cardiff Metropolitan University, University of Greenwich)

• Results due in December 2016

Implementation ViewTrends and tactics from the field

Jisc project in numbers

101 35 24 12 (+ 20)

Discovery activity assesses institutional readiness

– Goal: to assess institutional readiness (think organizational maturity)

» Measured on 26 factors crossing organizational and technical considerations

» Approximately 60% of the first 11 institutions are ready to implement Learning Analytics technology solutions

Source: Moving the Red Queen Forward, Educause Review September/October 2016, Dahlstrom

Varied activities show adoption flexibility

Profile Aim Activity Data sources

Russell Group Retention of widening participation + support for students to achieve 2.1 or better

Discovery + Tribal Insight + Learning Locker

Moodle + Student Records

Research led University Retention, improve teaching, empowering students

Discovery + OpenSource Suite + Student App

Moodle + Attendance+ Student Records

Teaching led University with WP mission

Retention - requirement to make identifying students more efficient so they can focus on interventions

Tribal Insight + Learning Locker

Blackboard + Attendance + Student Records

Research led University Student engagement Discovery + Student app + Learning Locker

Moodle + Student Records

Teaching Lead Understanding of how Learning Analytics can be used

Discovery + Technical Integration

Moodle

Organizational Trends

» Top level support is critical» Change culture makes things easier» Red tape is real (in policy management)» Academics looking for evidence-based results

It’s (almost) all about change management

& Tactics

» Convene a Learning Analytics committee (include students)» Identify champions and advocates» Adjust existing policies rather than creating new» Pilot the solution

It’s (almost) all about the data

& Tactics

» Perform data audits and quality checks (early and often)» Look for “all inclusive” offerings (predictions)» Look for integration options

Technical Trends

» Institutional infrastructure for data collection requires improvement» Unified data management desired but not realized» Data quality issues are common» Integration with existing infrastructure a challenge» Doing more with the same technical staff

Keep it simple, snowflakes

& Tactics

» Simplify for pilot; add complexity later» Overall, only add components you don’t already have» Flexibility (by institution and by vendors) is key

Pilot Trends

» Customizations are required to meet institutional needs» Ditto for integrations» Data gathering effort is considerable» Did we mention data quality?

Q&A

2/03/2016 The case for Learning Analytics

Interested in more detail?

» Data quality challenges» Predictive model research» Data collection, UDD, xAPI recipes, use of standards» Spark, ETL flows» ?

Michael Webbmichael.webb@jisc.ac.uk

Eitel J.M. Lauría, PhDeitel.lauria@marist.edu

Kate Valentikvalenti@unicon.net