the problems and promise of big data in advising...
TRANSCRIPT
TheProblemsandPromiseofBigDatainAdvising
AdrienneSewellDirectorofAdvisingforRetentionandSophomoreInitiativesChiefEditor,JournalofAdvisingUniversityDivisionIndianaUniversityBloomingtonasewell@indiana.edu
Whenitcomestodata,wearen'talwayssurewhatwearelookingat….
Expectations….
ItwillsearchlikeGoogle!
ItwillmakerecommendationslikeNetflix!
Itwillfindwho’satrisksowecanretainthem!
BigDatacansolveanything!
• “BIGdataissuddenlyeverywhere.Everyoneseemstobecollectingit,analyzingit,makingmoneyfromitandcelebrating(orfearing)itspowers.….Bycombiningthepowerofmoderncomputingwiththeplentifuldataofthedigitalera,itpromisestosolvevirtuallyanyproblem— crime,publichealth,theevolutionofgrammar,theperilsofdating— justbycrunchingthenumbers.”
BUT• “Bigdataispronetogivingscientific-soundingsolutionstohopelesslyimprecisequestions.”
Eight(No,Nine!)ProblemsWithBigData,GARYMARCUSandERNESTDAVIS,NewYorkTimes
Predictiveanalyticsystemsinadvising
• Thesesystemspromiseto“increaseretentionandgraduationratesatatimewhenoutcomesareunderscrutinyandfundingforadditionalacademicsupportishardtocomeby.”(Korn,2012)
Predictiveanalytics,datamining,patternrecognition,machinelearning
PedroDomingos (TheMasterAlgorithm:HowtheQuestfortheUltimateLearningMachineWillRemakeOurWorld)FirstStage• Computerswereprogrammedbyus• Programmerslookedatdataandmadehypothesis/tests/redologic/etc
SecondStage• Computersprogramthemselves• Google:searchresults• Netflix:predictswhatyouwouldliketowatch• Smartphoneslearnaboutus:typos,voicerecognition,routesonGPS,howyouwalk
Machinelearning
• Toteachcomputerstohavetheabilitytolearnwithoutbeingprogramed• tonotonlyhavethelogictoanswerquestions,butalsowhatthequestionsaretobeginwith
• Relyon“masteralgorithm”• Datamining• Logictreesandinversededuction
• SomeconsideratypeofArtificialIntelligence• Ourjobistomakesuretheyaredoingwhatwewant
• Input(settinggoals)• Output(yougetwhatyouaskedfor)
• TheDanger:Machineswillgiveuswhatweaskforandnotwhatwewant
Whataboutethicsandprivacy?
• Privacyissuesinmachinelearning• Machineskeepingtrackofeverythingwedo
• “Peopleworrythatcomputerswillgettoosmartandtakeovertheworld,buttherealproblemisthattheyaretoostupidandtheyhavealreadytakenovertheworld.”
PedroDomingos,TheMasterAlgorithm
Predictionsareonlyasgoodasthealgorithm
Whatifpredictionsarewrong?
• “Consistently,andusingsomeofthemostsophisticatedpredictiveanalyticstoolsintheworld…theworldwasallbutcertainoftheoutcomeofthe2016American election.”• “Whatwelearnedfromthiselectionisthatrelyingexclusivelyon“BigData”canmissthegoldoftenhiddeninsomemuchsmallerandmuchdifferentdatapockets.”
KarenWebster,7DeadlyDataSinsofthe2016Election
PredictiveAnalyticsinAdvising
• "Old-schooladvisingisaboutwhoappearsinfrontofyou—it’sverylimited,"saysRichardD.Sluder,ViceProvostforStudentSuccess."New-schooladvisingisusingpredictiveanalyticstotargetaspecificgroup.“
SpotlightonRetention:Studentscan’tgraduateiftheydon’treturn
EricHoover,ChronicleofHigherEducation
• Butisthistrue?• Advisorsdataset:experiencewithotherstudents• UniversityDivision:outreach/rosterreview(reviewofstudentrecords)foryears
StudentSuccessCollaborative(SSC)
• EducationAdvisoryBoard(EAB)
SSCPilotFall2013atIUBloomington
• Onlyafewunits• Riskandsuccessmarkersnotveryuseful• Filtersuseful
• Fall2014fullrelease• Createdexpectationsforuse• Incorporatedfiltersforoutreach“campaigns”• Trackedusage• Reportedissues/errors
• Fall2017cancel/notrenew• Whatdidwelearn?• Wheredowegofromhere?
Whatwasaimedtobesimplecameacrossassimplistic
Issues
• Blackbox:whatarethecorrelations?• Predictionsforallstudentsnomatterhowweakthecorrelation• Needforadvisors’input
Wheredowegofromhere?
• Whendevelopingsystemsacknowledgethatadvisorshaveknowledge• Advocateforadvisorinput/testingindevelopment• Recognizethatnotallpredictionshaveequalaccuracy• Askquestions,checkoutputs,adjustquestions
Advisorsasexpertsofthetext(studentrecords)
CourseRecommendercreatedbyBloomingtonAssessmentandResearch• Creatingpredictionsforcommonlytakencoursestopredictsuccessinotherclasses• Calculusgrade:canitpredictoutcomeinBiology?Businessclasses?
• Testingagainstadvisorknowledge• Whatcategoriesofinformationdoadvisorsusetoaccessrisk?• Doesdotheanalyticsanything?
• AnApplicationofParticipatoryActionResearchinAdvising-FocusedLearningAnalytics,StefanoFiorini,AdrienneSewell,MathewBumbalough,Pallavi Chauhan,LindaShepard,GeorgeRehrey andDennisGroth
PredictionAccuracy
• Onlyshowspredictionswithastrongcorrelation• Validactionableinformation• Workingtoavoidrateoffalsepositives
• Mostsystemsprovideapredictionforallstudents• Lessaccuracy
CourseRecommender
Behavior
Difficult course
GPA
Inconsistent performance
Prerequisite
Quantitative
Repeat course
Similar course
Test scores
Withdrawal Incomplete
AAAD-A380
ABEH-A200
ANAT-A215
ANTH-A122
ARTH-A101
ARTH-A102
AST-A105
BIOL-L112
BIOL-L113
BIOL-L211
BIOL-L311
BIOL-L312BIOL-L318
BIOL-L321
BIOL-M200
BIOL-M250
BIOL-M315
BIOL-M485
BIOL-P451
BIOL-Z406
BIOL-Z460
BUS-A200
BUS-A201BUS-A202
BUS-F300
BUS-G300
BUS-K201
BUS-L201
Behavior
CHEM-C101
CHEM-C117
CHEM-C127
CHEM-C341
CHEM-C342
CHEM-C343
CHEM-S343
CJUS-K300
CLAS-C101
CLAS-C206
CLAS-L250
CMLT-C110
COGS-Q320
COLL-C103
COLL-C104
COLL-P155
CSCI-A110
CSCI-A290
Difficult course
EALC-E110
ECON-E201
ECON-E202
ECON-E370
EDUC-P248
EDUC-Q200
EDUC-X150
EDUC-X158
ENG-L204
ENG-L389
ENG-W103
ENG-W131
ENG-W231
FINA-N198
FRIT-F150
FRIT-F200
GEOG-G109
GEOG-G306
GEOL-G105
GNDR-G225GNDR-G335
GPA
HISP-S105
HISP-S200
HIST-A383
HIST-H102
HIST-H206
HIST-J300
INFO-I101
INFO-I201
INFO-I202
INFO-I210
INFO-I308
INTL-I204
Inconsistent performance
LATS-L396
LSTU-L100
MATH-D116
MATH-D117
MATH-M118
MATH-M119
MATH-M120
MATH-M18
MATH-M211
MATH-M311
MATH-V118
MATH-V119
MSCH-C101
MSCH-C207
MSCH-C226
MSCH-J300
MSCI-M216
MSCI-M375
MUS-Z111
PHIL-P150
PHSL-P215
PHYS-P150
PHYS-P201
PHYS-P202
PHYS-P221
POLS-Y100
POLS-Y103
POLS-Y109
PSY-K300
PSY-P101
PSY-P102
PSY-P304
PSY-P324
PSY-P335
PSY-P337
Prerequisite
QuantitativeREL-A235
REL-R160
Repeat course
SOC-S320
SOC-S321
SOC-S339
SOC-S371
SPEA-E162SPEA-E311
SPEA-H124
SPEA-H402
SPEA-K300
SPEA-V160
SPEA-V161
SPEA-V220
SPEA-V236
SPEA-V246
SPEA-V252
SPEA-V261
SPEA-V370
SPEA-V372
SPEA-V406
SPH-C213
SPH-E311
SPH-H150SPH-I187
SPH-K150
SPH-M333
SPH-M382
SPH-M418SPH-N231
SPH-N331
SPH-R142
SPH-R200
SPH-R210
SPH-W147
SPH-Y277
SPHS-A200
STAT-S100STAT-S301
STAT-S303 STAT-S320
SWK-S300
Similar courseTest scores
Withdrawal Incomplete
StudentSuccess
• Analyticsisonlyonepieceinastudentsuccesssystem,whichrequires“commitmenttopersistent,personalizedactionsandinterventionstoimprovestudentsuccessguidedbyanalytics-basedinsights.”(BaerandNorris,2013)
Ethicsinassessingrisk
• Whendoourdatapointsbecomeethicalissues?• Whatabouteconomicbackground,financialneed,race,etc?
• Canassessingriskbecomeaself-fulfillingprophecy?• Approachesmatter
Dataalonewon’tsaveus
• Datacanhelpanswerquestions• Datacanhelpusdescribe/discoveranpattern• Datacanhelpfiguringoutstudentstoreachoutto
Advisingmustcontinuallyadjustandusedatawisely• Innovatewaysofusingdata• Usedatatoinformapproaches• Approachesarecriticalinstudentsuccess!
Creativeusesofdata
Wordusagepatternsfromaprobationexercise(Voyant)
Aid+money+financial+scholarship+scholarships=39
Parents+family+mom+mama+dad=62
Kicked(out)+dismissed+expelled=57
Grades+gpa =64
Life+future=38
In-housereports
Tableaureports
Dataiscriticallyimportantsoweneedto…
• Haveabasicunderstandingofwhatthedataweareusingmeans• Learnhowtouseitwiselyandcreatively• Askgoodquestions• Questionoutputswhentheydon’tmakesense
Data…
• Baer,L.L.,&Norris,D.M.(2015).Whateveryleaderneedstoknowaboutstudentsuccessanalytics(WhitePaper).Civitas Learning.
• Bumbalough,M.,Chauhan,P.,Fiorini,S.,Groth,D.Sewell,A.,P.,Shepard,L.,Rehrey,,AnApplicationofParticipatoryActionResearchinAdvising-FocusedLearningAnalytics,(unpublished)(2017)
• Burns,B.(2016,January29).BigData’sComingOfAgeInHigherEducation.Forbes.Retrievedfromhttp://www.forbes.com/sites/schoolboard/2016/01/29/big-datas-coming- of-age-in-higher-education/#14fd578a2a32
• Buyarski,C.,Murray,J.,Torstrick,R.,LearningAnalyticsAcrossaStatewideSystem,NewDirectionsforHigherEducation(Fall2017)
• Domingos,P.TheMasterAlgorithm: HowtheQuestfortheUltimateLearningMachineWillRemakeOurWorld (2015)
• Korn,M.(2013,October8).CollegesMineDatatoHelpStudentsStayonCourse.WallStreetJournalhttp://www.wsj.com/articles/SB10001424127887324123004579057200819823872
• Neff,G.(2013).Whybigdatawon’tcureus.BigData,1(3),117–123.http://doi.org/10.1089/big.2013.0029• MARCUS,G.&DAVIS,B.Eight(No,Nine!)ProblemsWithBigData,NewYorkTimes• Webster,K7DeadlyDataSinsofthe2016Electionhttp://www.pymnts.com/big-data/2016/the-seven-deadly-data-sins-of-the-2016-american-election-for-president/