@stevelonn

ActionableLearningAnalytics:PracticalLessonsforTeaching&Learning

StevenLonn,Ph.D.

@stevelonn

Acknowledgements

• ColleaguesinDEI atMichigan• ColleaguesandstudentsinLEDLab atMichigan

• DevonCancilla andteamatUMKC• StephanieTeasley• TimMcKay• MatthewPistilli• KimArnold• JoeWaddington

2

@stevelonn

Who(ifanyone)isusingthecontentyoucreatefor

yourcourses?

3

@stevelonn

IsThis“Actionable?”

TheChronicleofHigherEducationJanuary5,2016http://chronicle.com/article/This-Chart-Shows-the-Promise/234573

4

@stevelonn

JoannaMirecki Millunchick

• ProfessorofMaterialsScienceandEngineering

• TeacheslargeintroductoryMaterialsScienceandEngineering(MSE)course

5

@stevelonn

ScreencastsforHWSolutions

6

@stevelonn

Screencastusagepositivelycorrelatedwithcourseperformance

• Chemicalengineersusedscreencaststheleast,yetreceivedthehighestgrades

• Industrial&operationsengineersusedscreencaststhemost,andreceivedaveragegrades

7

@stevelonn

What’sgoingoninthecurriculum?Materials Science&Engineering

• Physics ofMaterials

• PrinciplesofEngineeringMaterials

• ThermodynamicsofMaterials

• KineticsandTransport

• StructuresofMaterials

• MaterialsLaboratoryIandII

ChemicalEngineering

• IntrotoMaterialandEnergyBalances

• ChemicalandEngineeringThermodynamics

• FluidMechanics• MassandHeat

Transfer• Separation

Processes• Reaction

EngineeringandDesign

• ChemicalEngineeringLaboratory

AerospaceEngineering

• IntrotoAerospaceEngineering

• IntrotoAerospaceEngin.Systems

• IntrotoSolidMechanicsandAerospaceStructures

• IntrotoGasDynamics

• AircraftandSpacecraftStructures

• Aerodynamics• Propulsion

Industrial&Operations Eng.

• Economic DecisionMaking

• OperationsModeling

• Probability andStatistics

• OptimizationMethods

• MarkovProcesses• Ergonomics• LinearStatistics• DataProcessing

Screencastsshouldhaveagreaterimpactasasupplementaryresourcetoaidstudentlearningforstudentswithlessfamiliaritywiththecoursecontent.

8

@stevelonn

CaseStudyonExamPerformance

• IfIOEstudentswatchedany screencasts,theyperformedbetteronaparticular question.– r =.425p =.001

• IfIOEstudentswatchedaparticular screencast,therelationshipevenstrongerfortheperformanceonthatparticular question– r =.375p =.003

9

Green,Pinder-Grover&Millunchick,2012

@stevelonn

Summary• Analytic:LMShitsandperceptionsurveysaboutsupplementaryproblem-setscreencasts

• Finding:Sig.content-performanceimpactforlearnerswithoutpriorMSEknowledge– NoimpactforstudentswithpriorMSEknowledge

• Whatcanyoudo?:Beginbybreakingdownassessmentsbydiscipline/sub-discipline.Unpackanysig.differences– (gender,ethnicity,curriculum,etc.)

10

@stevelonn

WhatisLearningAnalytics?

11

@stevelonn

Learninganalyticsisthemeasurement,collection,analysisandreportingofdataaboutlearnersandtheircontexts,forpurposesofunderstandingandoptimisinglearningandtheenvironmentsinwhichitoccurs.

1212

@stevelonn

measurement,collection,analysisandreporting

ofdata

1313

@stevelonn

learners andtheircontexts

1414

@stevelonn

understanding andoptimising learning

1515

@stevelonn

IsUMKCpreparedtoimplement

learninganalyticstoday?

16

@stevelonn

WhenisanInstitutionReady toImplementLearningAnalytics?

17

Read�i�ness(ˈred-ē-nis)

noun1 willingness to do something.

2 the state of being fully prepared for something.

3 immediacy, quickness, or promptness.

@stevelonn 18

TheNeedforReflection

• Comprehensiveunderstanding

• Multipleperspectivesneeded

• Multiplelevelsneeded

TheLearningAnalyticsReadinessInstrument

Arnold, Lonn&Pistilli,2014

@stevelonn

LARIBetaParticipants

19

Assoc/Pub-R-L,Bac/A&S,&Bac/Diverse

Master’sLarge&ResearchUniversities(HighResearchActivity)

ResearchUniversities(VeryHighResearchActivity)

5institutions129respondents

15institutions337respondents

4institutions84respondents

@stevelonn

ReadinessFactors

20

Factor # SurveyItems

Example

Culture 22 “Myinstitutionhasaculturethatacceptstheuseofdatatomakedecisions.”

DataManagementExpertise

8 “Myinstitutionhastheabilitytostoreincreasinglylargevolumesofdata.”

DataAnalysisExpertise

6“My Institutionhasprofessionalswithmathematical/statisticalexperienceinmanipulatingandtransforming data.”

Communication&Policy Application

7 “MyInstitutionhasprofessionalswithbusinessacumeninmarketing/publicity.”

Training 3“Myinstitutionhasprofessionalswithcustomer-facingsupport experienceintrainingconstituentsontheuseofnewsystems.”

Oster,Lonn,Pistilli,Brown,2016

@stevelonn

CultureisPervasive

21

@stevelonn

GrowinganInstitutionalLearningAnalyticsCommunity

22

“StudentLearningandAnalyticsatMichigan”seminarseries

crlt.umich.edu/slam

Othercomponents:

• LearningAnalyticsfacultytaskforce(3yearcharge)• “ExploringLearningAnalytics”researchprojects• Learninganalyticsfellowsprogram• Streamliningdataaccessandestablishingnewdatasets

@stevelonn

PracticalLearningAnalyticsMOOC

23

Everyoneineducationhasquestions….PLAisaboutansweringthem.

ReturningSpring2016

@stevelonn

Summary• Analytic:Institutionalreadinessmeasuredbyinstitutionalreflection(surveyresponses)

• Finding:5factors:Culture,datamanagement,dataanalysis,communication&policyapplication,&trainingallimportantforreadiness

• Whatcanyoudo?:Helpestablishcommunicationaboutlearninganalyticswithcolleagues.– e.g.,Organizeamonthlyseriestodiscussandhavespeakers.

24

@stevelonn

Aretheredifferentkindsofanalyticsineducation?

25

@stevelonn

Howlonguntiltheygraduate?

4-yeargraduationrate:22%6-yeargraduation rate:51%

Whichstudentsareaccepted?

Roughly4,300 HSstudentsapplyUMKCaccepts65-68%ofapplicants

Ofthose,about39%enroll(F15 n=1,021)

Whichstudentsstay?

Freshman retentionrate:73%Part-timestudentretentionrate:28%

LearningAnalyticsvs.AcademicAnalytics

26

@stevelonn

LearningAnalyticsvs.AcademicAnalytics

27

AcademicLoad

EnrollmentbyAcademicLevel

EnrollmentbyDistanceEducation

Sources:irapweb.umkc.edunces.ed.gov

@stevelonn

HowtoMeasureStudentLearning

• Traditionalmeasures

• Holisticmeasures

• Psychometrics

• 21st CenturySkills

28

@stevelonn

Data+Tools=NewWaystoMeasureTeachingandLearning

29

Student InformationSystemsLearningManagementSystems

Onlinehomework, peerevaluation,discussions, practicetools,etc.

DashboardsStatisticalanalysistools

MachinelearningSocialnetworkanalysis

Naturallanguageprocessing

@stevelonn

NewWaysforInstructorstoViewStudentActivity&Performance

30

@stevelonn

UnpackingCourseResourceAnalytics

31

• NancyKerner• LecturerIV,GeneralChemistry(Emeritus)

• TaughtlargeintroductoryChemistrycourse

@stevelonn 32

• 10semestersofdata– 8,762students;703,191“hits”

• 4categoriesemerged– Courseinformation– Lecture– Assignments– ExamPreparation

UnpackingCourseResourceAnalytics

@stevelonn 33

• Raw#of“hits”:Astudents>Bstudents>Cstudents

• 25th à 50th percentileofresourceuse:

UnpackingCourseResourceAnalytics

ResourceCategory ‘A’vs.‘C’ ‘B’vs.‘C’

CourseInformation 0.856 0.907

Lecture 1.228 Not SignificantAssignments Not Significant Not Significant

ExamPreparation 1.811 1.565

Waddington &Nam,2014

@stevelonn

DoestheModelScale?HowCanitScaleforYou?

34

• Expandedto7largeSTEMcourses– 24Kstudents

• Similartrends– OnlyExamPrepresourcesrelatedtobetterperformancein5of7courses

Waddington, Nam,Lonn&Teasley,InReview

@stevelonn

Summary• Analytic:LMShitsoncourseresourcesandsemi-manualcategorizationinto4keytypes.

• Finding:Exampreparationmaterials(whenclicked)onlysignificantresourcetyperelatedtoimprovedcourseperformance.

• Whatcanyoudo?:IdentifythetypesofcourseresourcesavailableinyourLMSwebsite– whatcategoriesemerge?– WhatdoestheLMStellyouabouthow,andwho,isusingdifferentcategories?

– Whatistheeffectonlearning?

35

@stevelonn

Whatlearninganalyticsinnovationsareonthehorizon

atMichiganandbeyond?

36

@stevelonn

PresentingCourse-RelatedAnalyticstoStudents

37

@stevelonn

UsingAnalyticstoCoachStudents

38

@stevelonn

NaturalLanguageProcessing

39

• Outofthebox:Sentiment,bag-of-wordsgroupings,readinglevel• Pairs,triads,sentencestructures• Topicmodeling• Customdictionariesbydiscipline,contentarea,etc.

@stevelonn

EthicalConsiderations

40

@stevelonn

AnalyticsforLearners

41

@stevelonn 42

Knowingis notenough;wemustapply.

Wil l ingis notenough;wemustdo.

―JohannWolfgangvonGoethe

@stevelonn 43

ThankYou!

StevenLonn,Ph.D.AssistantDirectorforAssessment&EvaluationDigitalEducation&InnovationUniversityofMichigan

slonn@umich.edu@stevelonn

digitaleducation.umich.edu