Martin Pusic MD PhD on behalf of…Institute for Innovations in Medical Education

NYU School of Medicine

Observe-Interpret-Act:

An NYULMC Learning Analytics Case Study

LEARNING

ANALYTICS“Learning analytics refers to the interpretation of a wide range of data produced by and gathered on

behalf of students in order to assess academic

progress, predict future performance, and spot

potential issues”

- DOE 2012

How is Big Data Different?Traditional Big Data

Data Collection Purposeful Incidental / Opportunistic

Intrusiveness High Low

Data Acquisition Cost High Low

Frequency e.g. Quarterly Continuously

Temporality Static Reports Dynamic Dashboards

Hypotheses Causal Associations

Sample Size Small biopsies Large swaths

NYU Education Data Warehouse

Education Data

Warehouse

LMS

Curricular

Content

AdmissionsExams

ePortfolioPatient

Log

Learning

ModulesEvaluations

SIS

Reporting and Analytics

Data Marts for

Education Research

Simulation

Clinical Data

Warehouse

NYU Practice

NetworkEpic EMR

iBeacons

New “Listeners”

• iPads

• Point-of-care digital forms

• Immediate Lecture evaluations

• Learning interactions with EHR

• Anything a SmartPhone can do

NYU Education Data Warehouse

Education Data

Warehouse

LMS

Curricular

Content

AdmissionsExams

ePortfolioPatient

Log

Learning

ModulesEvaluations

SIS

Reporting and Analytics

Data Marts for

Education Research

Simulation

Clinical Data

Warehouse

NYU Practice

NetworkEpic EMR

NYU Education Data Warehouse

Education Data

Warehouse

LMS

Curricular

Content

AdmissionsExams

ePortfolioPatient

Log

Learning

ModulesEvaluations

SIS

Reporting and Analytics

Data Marts for

Education Research

Simulation

Clinical Data

Warehouse

NYU Practice

NetworkEpic EMR

Gartner Model

© Gartner Group

Path Diagram for the Class of 2014

• Path coefficients (with standard errors) were calculated using multiple regressions.

• Each variable is set as the dependent variable and all variables to its left as possible predictors.

• We only include the paths that are statistically significant.

• Think of each number in the following way – adjusted for all the other incoming arrows, a one standard deviation change in the predictor results in an x standard deviation change in the thing being predicted.

• For example, the regression equation predicting USMLE1 includes MCAT, College GPA and Medical School GPA. College GPA does not signfiicantly predict USMLE1 so there is no number. Adjusted for MCAT score, a one standard deviation rise in Med_GPA results in a 0.62 standard deviation rise in USMLE1 score.

MCAT

College GPA

USMLE1

0.26(0.067)

0.37(0.085) 0.62(0.073) USMLE2SHELF

0.36(0.09)

0.42(0.09)

0.29(0.09)

0.57(0.08)Med_GPA

NYU Analytics Center

Clinical Curriculum

High level

view for

chairs

Curriculum Stage

LEARNING

ANALYTICS“Learning analytics refers to the interpretation of a wide range of data produced by and gathered on

behalf of students in order to assess academic

progress, predict future performance, and spot

potential issues”

- DOE 2012

LEARNING

ANALYTICS“Quality improvement refers to the interpretationof a wide range of data produced by and gathered

on behalf of clinicians in order to assess progress,

predict future performance, and spot potential

issues”

- DOE 2012

Quality

Improvement

The New CME

• Un-Announced Standardized Patients

• In-Situ Simulations

• Patient Reported Outcome Measures

• Process Metrics

– E.g. Door to Needle Time in Stroke Activations

– E.g. Surgical Video

How is Big Data Different?Traditional Big Data

Data Collection Purposeful Incidental / Opportunistic

Intrusiveness High Low

Data Acquisition Cost High Low

Frequency e.g. Quarterly Continuously

Temporality Static Reports Dynamic Dashboards

Hypotheses Causal Associations

Sample Size Small biopsies Large swaths

Feedback

Door to Needle Time

Discussion