LearningAnalytics:HarnessingDataSciencetoTransformEducation

TimMcKay:UniversityofMichigan@TimMcKayUM,Blogat21stCenturyHigherEd.com

TheUniversityofMichigan

• 200yr oldpublicresearchintensiveuniversitywith19SchoolsandColleges&6,800faculty

• Highlyselectivegroupof29,000undergraduateand16,000graduatestudents

• Annualbudgetof$7billion,including$1.4billioninfederallyfundedresearch(#1public)

Particleastrophysics:Carefullymeasuringnothing…

ObservationalCosmology:MeasuringEverything…

Thisisobservationalscience:drawinginferencewithoutclassicalmethodsofexperiment…

Teachingthousandsofstudentsintroductoryphysicsofeverykind…

8yearsasDirectorofLSAHonorsProgram,caringfor2000students

acrossthedisciplines…

Myneedfordataaboutthesestudents&theirexperiencesledtocontinuallyexpandingeffortsin

LearningAnalytics

The20th Centurybeganwithanindustrialrevolution.Publichighereducationjoinedin:explodinginscaleandadoptedbureaucratic,industrialapproaches,including

standardizedtests,credithours,GPAs,majors,andminors.

Toooften,wepursuea20th century,industrialformofoptimization:seekingasinglesystemwhichmaximizeslearningacrossapopulationofstudents.

The21st Centurybeganwithaninformationrevolution.Weknowmoreaboutourstudentsthanweeverhaveandconnectthemwithus,information,oneanother,andtheworldinunprecedentedways.

Ourgoaltodayisa21st century,informationageformofoptimization:adaptingthesystemtoindividuallyoptimizelearningforeachstudent.

Rememberthegoal…

1984

Whyhasn’tthishappenedalready?

Whysomuchvisceralresistancetotheideaofusingtechnologyto

personalizeeducation?

• >98%oftheseare:• Small&changeable• Taughtinidiosyncratic,engaged,andcreativeways

• <2%ofthemare:• Large&relativelystable• Taughtinindustrial,remote,andtraditionboundways

Onereason?Therearetwowaystoteachalargenumberofstudents…Wehave9200coursesatUM

Whypersonalize?Forthe735studentsinmyclass…

Nita

Frank

Weareoftenfooledbyafalsesenseofpersonalization.Thesetwostudentsare0.3%ofthe

wholepool…

FocusonEducation@Scale• EvenatbigUniversities,mostcourseshavewhattheyneedtobeexcellent.Largeintroductoryandmanyonlinecourses don’t– Theycouldbedramaticallybetterwithadifferentapproachtoinstructionandprofessionalsupport

• Improvingeducationatscaleisamajorsociotechnicalchallenge, requiringboth:– Newinformationtechnologyforpersonalization– Newsocialnormsforcoursedesign&delivery

It’shardtobeatBloom’stutorwhereshecanact.Learninganalyticsshouldfocusonimprovingthingswhereshecannot.

Fivethemesofourwork

Learninganalyticsforpersonalization

1. Ethics:whatarewedoingandwhy2. Measurement:datacollectionandmanagement3. Analysis:modeling,extractionofmeaning,

learningfromtheexperienceofall4. Action:decisionmaking,storytelling,creating

themotivationforchange5. Synthesis: Buildingalearninglaboratory

#1Ethics:Whatwe’redoingandwhy

Informationethics:agrandchallenge

• Whatprinciplesshouldgoverncollectionanduseofdataaboutindividualsineducation?– Whatdataisrelevantforeducation?– Normsofconsent,privacy,autonomy?Howaretheydifferentwithinanacademiccommunity?

– Howareexperimentsineducationalpracticerelatedtoresearchnorms?

• WeneedtoensurethatcommercialEd-Techisapartof(&constrainedby)thisconversation

SixAsilomar principles

1. Respectfortherightsanddignityoflearners:transparency,consent,protectionofprivacy

2. Beneficence:maximizebenefits,minimizeharm3. Justice:benefitall,reduceinequalities4. Openness:learningandresearcharepublicgoods5. Thehumanityoflearning:insight,judgment,&discretionare

essential,weshouldkeeplearninghumane6. Continuousconsideration:ongoing,inclusivediscussionof

changingethicalcircumstances

http://asilomar-highered.info/

Wheretohavethisconversation?

Example1:PredictiveModeling

Aseducators,welearnfromthepastinordertochange thefuture…

Example2:ProbingInequity

Toooften,weassumethatidenticaltreatmentmodelsfairness…

Example3:MeasureableTypes

• Dataoftenusedtocategorize,collectingindividualsintogroups

• Theselabelsareoftenreductiveandinvisibletolearners

• Always incomplete,substitutingacategoryforindividuals

• Cheney-Lippold (2017)‘measureabletypes’≠complexsociallyconstructedclasses– gender≠‘gender’– text≠‘positive’

• Categorizationstoooftenonedimensional,excludingintersectionsofidentity

Howtolimittheimpactofreflexivelyreductivedatarepresentation?

IntersectionalityinLA

• Learnmethodologyofintersectionalityfromfeministscholarship*:usemultipleapproaches– Intercategorical:focusonvariationacrosssociallyconstructedprovisionalcategories

– Intracategorical:analyzevariationw/incategories– Anticategorical:realpersonalization,nocategories

• Betransparent,allowforagency:resistlabellingindividualsw/ounderstandingandconsent

*McCall,L.(2005).Thecomplexityofintersectionality.Signs:Journalofwomenincultureandsociety,30(3),1771-1800.

#2Measurement:Datacollectionandmanagement

Whatdowemeasure?• Whatwemeasurenow:

– Admissionsinformation– Coursetaking&grades– Degrees&honors

• Whatwe’restartingtorecord(explosivegrowth)– Processoflearning:

clickstreams,discussions,video,coursestructures

– Productsoflearning:forumposts,essays,papers,presentations,theses

• Whatwewanttohave:Detailed,relevant,evolving

portraitsofeverystudent'sbackground,interests,goals,andaccomplishments

• Theseportraitsshouldbeusedtohelpstudents,faculty,administrators,staffbetterunderstandhighereducation

Justforstudentrecords,thereare157pagesofdatadescription…hundredsoforganicallyevolving,interactingtables…

1stchallenge:Datacleaning&aggregation

Examplepartialsolution:UMLearningAnalyticsDataArchitecture

A‘regularrelease’modelforcleanresearchdata.SimilartothoseinopenscienceprojectsliketheSDSS

orGAIAspacemission

Howtoreleaseinformation whileprotectingprivacy?

• Asmuchaspossible,weshouldleteveryonelearnfromtheexperienceofall– Restrictedreportingtools– accesstodigestedinformation,withintools(Ex:ART2.0,ECoach)

– Existingresearchprotocols– IRBoversight,anonymization=>theLARCapproach

• Newapproachesareemergingindatascience:syntheticdatacontainalltheinformationbutnoneofthedetails

• Personalprivacycan(&must)beprotectedwell.Weshouldrethinkinstitutionalprivacy…

Bettermeasuresoflearning• Grades:performance

measuresofunrecordedtasks,meanttoestimateunknownoutcomes,quantifiedonill-definedscales

• Weshouldbemeasuringlearning– increasesinwelldefinedknowledgeandskills– andfocusingonindividualgrowthovertime

• Direct:preandposttestingalignedwithlearninggoals.Goodforfoundationalcourses?

• Indirect:DataSciencetoolsforextractingmeaningfromproducts– Simple:IRT,topicmodeling

andbeyond– Complex:peerevaluation,

NLP,directrepresentationratherthandatareduction

IntellectualBreadth DisciplinaryDepth RangeofExperience

Engagement&Effort Social&ProfessionalNetworks

AcademicPerformance

Measuringwhatmatters:theTranscriptoftheFuture

Studentsconnectthroughcourses

Coursesconnectthroughstudents

Canwequantifyintellectualbreadth?

Exploreeachstudent’snetworkofconnection

• Courseco-enrollment:wellmeasured,largebipartitenetwork

• Betterrepresentationsofinteractioncoming

Comparemeasurednetworkstructurestoappropriaterandomgraphs– measurediversityofconnection

Exposesisolationofmajors,allowscomparisonofindividualswithinamajor

ConnectedinmajorConnected

outofmajor

#3Analysis:Learningfromexperience

Methodsforreliableinferencefromobservationaldata

Threeexamplesusingdifferentmethodologies:

1. Areourclassroomsequitable?2. Dolearningcommunitieswork?3. Areplacementexamsusedwell?

BTEWTE

#1:Koester/Grom/McKayAreourclassroomsequitable?

Studentperformanceisinfluencedbybackgroundandpreparation.

Forexample:gradesinphysicsrelatedtogradesinothercourses.

ObservedcorrelationClassroomequity

Genderedperformancedifferences

<GPA– Grade>Male=0.32<GPA– Grade>Female=0.59

GPD=0.27

Theseperformancedifferencesremain

whenweaccountforallmeasuresofbackground

&preparation.Unexplainedperformancedifferenceslikethisaresignsofclassroominequity.Wemustlookforandaddressthesedisparateimpacts.

Koester,Grom,McKay:https://arxiv.org/abs/1608.07565

Classroomequity

AlllargeintroSTEMlecturecourses+Econ101/102

Measuredofgradepenalty&GPDacrossalllargecoursesatUM:Strikingpatternsofgenderedperformancedifference

Classroomequity

Datafrom2000– 2012foralllargeSTEMlectureandlabcourses

Labcourses

Lecturecourses

Details,includingtestsofmanyotherpossible

performancepredictors:arXiv 1608:07565

Intercategorical complexity

Classroomequity

Biology Chemistry

Math&Stats Physics

DatafromfiveBig10Schools:SimilarGPDpatternsacrosslecture&labSTEMcourses.

Matz etal. AERAOpeninpress

Thisanalysisusesbothhierarchicallinearmodelingandquasi-experimentalmatchingmethods

#2:Brooks/Morgan/Maltby - HSSPImpact

LivingLearning

Quasiexperimental design

HealthScienceScholarsProgram

Exampleresults

HSSPsignificantlyincreasedthelikelihoodofBSandadvanceddegreesforunderrepresentedand

first-generationstudents.

LivingLearning

Quasiexperimental design

Michigan1:2:1IntroductoryChemistryCurriculumModel:

Traditional2:2IntroductoryChemistryCurriculumModel:

2 Semesters General Chemistry 2 Semesters Organic Chemistry

Chemistry130

Chemistry210&215 Chemistry230

#3:Shultz/Gottfried/WinschelChemistryPlacementAnalysis

ChemPlacement

Regressiondiscontinuity

HowdoestakingGenChem firstmatter?

ChemPlacement

Shultz,GingerV.,AmyC.Gottfried,andGraceA.Winschel. JournalofChemicalEducation 92.9(2015):1449-1455.

Regressiondiscontinuity

Whatcangowrongwithallofthesemethods?

Educationisharderthanphysics…Replicability≠ generalizability

Action:Puttingdatatowork

Decisionmaking,storytelling,motivatingchange

Howtoputdatatowork…

Aspectrumofinformationagency…

Givestudents,advisors,facultythedata~directly.Letthemdecidewhattodo.

Give‘experts’thedata.Havetheminterpret,andmakedecisionsforstudents,advisors,

faculty.

Givethedatatoboth!Haveexpertshelpstudents,

advisors,facultyinterpretdata:shapedecisionsusingbehavioralscience,choice

architecture,nudges

Information&advisingsystemsalwaysfaceaspectrumofagency.What’snewisthe

richnessofinformationandanalysis.

Todothis,youneedtoolswhichprotectprivacy

whilesharinginformation,professionallydesigned

fortheirusers.

UMDigitalInnovationGreenhouse

Takegoodideasdevelopedoncampusfrominnovationtoinfrastructure,supportEd-TechR&D,personalize

educationatscale

UniversityTeaching

Community

UNIZIN

Startups

ExternalResearchFunding

ResearchFindings&

Pubs

Innovators&pioneeringadopters

DIGteamofDevelopers,

U/XDesigners,BehavioralScientists

Communitiesofpractice:faculty,students,staff

UniversityResearch

Community

UniversityIT:supportat

scale

DIG:ahomeforacademicR&D

DIGwasborninMay2015:Aplace,ateamofinnovators,

originallyinDEILabonWashington

DIGhasgrown,andnowlivesatopourlibrary

DIGteamconnectsfaculty/staff

FACULTYDIRECTORTimMcKay

OPERATIONSDIRECTORMikeDaniel

FACULTYCHAMPIONSGusEvrard(LSA)BarryFishman(SI)ElisabethGerber(Ford)AnneGere(Sweetland)TimMcKay(LSA)PerrySamson(ENG)GingerSchultz(LSA)

LEADBEHAVIORALSCIENTISTHollyDerry

LEADDEVELOPERSBenHaywardCait HolmanKrisSteinhoffChrisTeplovs

LEADINNOVATIONADVOCATEAmyHomkes-Hayes

BEHAVIORSCIENTISTCarlyThanhouser

DATASCIENTISTKyleSchulz

DEVELOPERSDaveHarlanKushank RaghavOliverSaundersKe Ye

UX&DESIGNMarieHooperKristinMillerMikeWojan

Plus15-20studentfellowsdrawnfromComputerScience,SocialPsychology,Art&Design,UIDesign,BehavioralScience,

Education,&more….

Studentsareourbestcreativeengine:Fellows,

DesignJams&Hackathons!

DIGprojects:Arapidlygrowingportfolio

Andmore….

Providinginformationtoindividuals

Learningaboutclassesandmore:ART2.0

ART2.0– informationtoall

Coursecardswillbejoinedbyreportsoncoursesofstudy(majorsandminors)andpeople(students,faculty),alongwithtoolsforcurriculumexploration…

Expertinterpretationandadviceatscale

ECoach:computertailoredelectroniccoachingforequityandstudentsuccess

Experttailoredcommunication

• Builton20+yrs ofdigitalhealthcoaching• Aggregatesrichstudentinfofrommanysourcestotailorfeedback,encouragement,&advice

• Tailoringonbothwhat tosay,howtosayit,whospeaks:w/testimonialsfrompeers,etc.

• Allcontentwritten&testedbybehaviorchangeexperts,facultyfromdisciplines,students

• Atoolforhumanepersonalizationw/studentagency:allowingustospeak,sharedata,connect

ECoach:richlytailoredmessaging,informedbybehavioralscienceanddisciplinaryexpertise

ECoachisexpandingtomoreclasses,launchingatotherinstitutions,andsupporting

richarrayofresearchprojectsw/externalfunding.

Thisfall:8000studentsStats250EECS183,280Chem 130Physics140Bio171Econ101Engr 100,101ALA125FirstYear (6800more…)UCSantaBarbara

ECoachFuture ECoachsupportsresearchaddressingwidespreadgenderedperformancedifferences

inSTEMlecturecourses

ThisinterventionlaunchedthisfallasanRCTwithmorethan1000studentsineachofthetreatmentandcontrolarms.Firstofaseriesofupcoming

experimentsdeliveredinECoach

LearningtorespondtostudentwritingusingNLPetc.isa

majorgoalforthecomingyear.

Synthesis:Creatingalearninglaboratoryfor

studyingeducationatscale

FocusonFoundationalCourses• Large,relativelystable,

mostlyintroductorycourses• Servestudentswith

especiallyvariousbackgrounds

• Servestudentswithespeciallyvariousinterestsandgoals

• Foundationalcourseswhereweeducateatscaleareidealenvironmentsfortheapplicationofanalytics

• Bestlargecoursestaughtinmultigenerationalteamswithrolespecialization

• Rolesinclude:– Coursemanagement– Deliveryofinstructionon

largeandsmallscales– Instructionaldesign– Technology– Assessment&analytics– Studentsupport

• Coursesshouldbebroadly‘instrumented’forstudy

These‘foundational’coursesexistacrossmanydisciplines,mostofwhichareoutsidethenaturalsciences,sothisinitiativeiscampus-wide.

ALearningHigherEdSystem

DigitalInnovationGreenhouse

Largecourseteamdevelopsandsupportsatechnicalinfrastructureforresearch– gatheringdataandimplementinginterventionstudies

FoundationalCourseInitiative

CCDprocessprovidesasocialinfrastructureforsustainedresearchanddevelopment–practicegeneratingresearch

DataandTechnology People&SocialSystems

TranslationalResearchonEducationatScale:

TheFCIandDIGarecreatingthesociotechnicalframeworkneededtosupportrichtranslationaleducationresearch.

WearebringingresearchteamsintothisLearningLaboratory,intimatelyconnectingresearch&practiceintheauthentic,evolvingenvironmentofour

foundationalcourses.

Fiveyearsfromnow…

• Carefullydesignedandinstrumentedfoundationalcoursesestablishedasoneofthekeyelementsofalearninglaboratory

• Awellestablishedprogram,with~20-30coursesestablishedaspartofthislaboratory

• Weaimtoplayanimportantroleinestablishingarobustevidence-basisforlearninganalytics&personalizationatscale

Educating@scalein21st century• Teachingatscaleinthe

informationageaffordsunprecedentedopportunitiesforpersonalization

• Realizingtheseisamajorsociotechnicalchallenge

• WeareaddressingboththesocialandtechnicalchallengesassociatedwiththistaskatMichigan

• Ourcampusiscreatingalaboratoryforlearningatscale,connectingeducationresearch&practice

• Thiswillbecomea“learninghighereducation”communityinwhichwelearncontinuouslyfromexperienceincontext

• Itcanallbedoneinastudent-centeredwaywhileprotectingprivacy