sheila policy framework: informing ins9tuonal strategies and pol...
TRANSCRIPT
SHEILA policy framework: informing ins9tu9onal strategies and pol-icy processes of learning analy9cs
Yi-ShanTsai TheUniversityofEdinburgh
OldCollege,SouthBridge,EdinburghEH89YL UK
PedroManuelMoreno-Marcos UniversidadCarlosIIIdeMadrid AvenidaUniversidad,30,28911Le-
ganes,Madrid Spain
KairitTammets TallinnUniversity
Narvamnt25,10120Tallinn Estonia
KaireKollom TallinnUniversity
Narvamnt25,10120Tallinn Estonia
DraganGasevic TheUniversityofEdinburgh
OldCollege,SouthBridge,EdinburghEH89YLUK
MonashUniversity Scenic Blvd, Clayton VIC 3800
ABSTRACTThispaperintroducesalearninganalyticspolicydevelopmentframework developed by a cross-European research projectteam – SHEILA (Supporting Higher Education to IntegrateLearningAnalytics),basedoninterviewswith78seniorman-agers from51Europeanhighereducation institutionsacross16countries.TheframeworkwasdevelopedusingtheRAPIDOutcomeMappingApproach(ROMA),whichisdesignedtode-velopeffectivestrategiesandevidence-basedpolicyincomplexenvironments.Thispaperpresentsthreecasestudiestoillus-trate the development process of the SHEILA policy frame-work,whichcanbeusedtoinformstrategicplanningandpol-icy processes in real world environments, particularly forlarge-scaleimplementationinhighereducationcontexts.
CCSCONCEPTS• Security and privacy~Social aspects of security and privacy • Applied computing~Education
KEYWORDSLearning analytics, policy, higher education, strategy, ROMAmodel
ACMReferenceformat:Y.-S.Tsai,P.M.Moreno-Marcos,K.Tammets,K.Kollom,andD.Gasevic.2018.SHEILApolicyframework:informinginstitutionalstrategiesandpolicy processes of learning analytics. InProceedings of the Interna-tional Conference on Learning Analytics and Knowledge, Sydney, Aus-tralia, March 2018 (LAK’18), 11 pages. DOI:https://doi.org/10.1145/3170358.3170367
1 INTRODUCTIONANDBACKGROUNDHigherEducationInstitutions(HEIs)areconstantlycollect-
inglargeamountsofdataintheformofstudents’digitalfoot-printsduringtheirstudies.AlthoughHEIsstrivetoincreasethequality of teaching and learning by exploiting the collecteddata,thereareoftenbarriersthatpreventdatafrombeingusedsystematicallyandeffectively.Forexample,dataquality,own-ershipandaccess,organisationalculture,andexpertiseavaila-bletoimplementlearninganalytics(LA)areprevalentissuesthatneedtobeaddressedbeforeimplementation[4].Accord-ingtoFergusonandothers[10],althoughfundingopportuni-tiesforLAresearchandactivitieshaveincreased,thereisstillalackofsystematicandlarge-scaleimplementationsofLAinhighereducation.ThepreliminaryfindingsofaEuropeanpro-ject–SHEILA(SupportingHigherEducationtoIntegrateLearn-ingAnalytics)havedemonstratedthatnumerousHEIs inEu-ropeareeitherobservingthedevelopmentofLAorhaveen-gagedwithitpracticallywithoutadefinedstrategyormonitor-ingframeworktoensuretheeffectivenessandlegitimacyofLA
Permissiontomakedigitalorhardcopiesofpartorallofthisworkforper-sonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationonthefirstpage.CopyrightsforcomponentsofthisworkownedbyothersthanACMmustbehonored.Abstractingwithcreditispermitted.Tocopyotherwise,orrepublish,topostonserversortoredistributetolists,requirespriorspecificpermissionand/[email protected]'18,March7–9,2018,Sydney,NSW,Australia©2018AssociationforComputingMachinery.ACMISBN978-1-4503-6400-3/18/03…$15.00
LAK’18, March 2018, Sydney, Australia Y.-S.Tsaietal.
2
practices [30]. Drachsler and Greller identified uncertaintiesamonginstitutionsaboutlegalboundariesandethicallimitsre-gardingtheuseofpersonaldataforLA,inadditiontoprevalentfearofnegativeconsequencesfromtheapplicationofLA[8].Asaresult,theyproposedtheDELICATEchecklisttorebrandtheprivacyburdenthattheLAcommunitycarriedwithaqualitylabel.OtherfamousmodelsthatexisttoguidetheadoptionofLAcanbefoundinJisc’sCodeofPractice[5]andtheOpenUni-versity’s“PolicyonEthicaluseofStudentDataforLearningAn-alytics” [28]. However, these ethical and privacy guidelinesmaynotalwaysapplytoeveryinstitution’sownuniquecon-text.Inlightoftheneedforasoundpolicythatistailored`tomeetindividualinstitutions’uniquecontextsandensuresare-sponsibleandeffectiveuseofstudentdataforLA,theSHEILAprojectwaslaunchedwiththegoaltoassistHEIstodevelopin-stitutionalpoliciesforLA.Todoso,theprojectwillproduceapolicyframework(addressedastheSHEILApolicyframeworkhereafter)byengagingendusersofLAdirectlytounderstandtheir perceptions, expectations and concerns, as Knight andothers[15]havesuggestedthatusersareinthemostaccuratepositiontoidentifytheirownneedsandtoindicatehowtheirpracticescanbesupportedandimprovedbeforesolutionsaredesignedandimplemented.[20].Withdatacollectedfromthedirect engagement with stakeholders, the project team hasusedtheRAPIDOutcomeMappingApproach(ROMA)toscopeexistingpracticesofLAamongHEIs inEurope,and tomakesuggestionsforpolicydevelopment.AlthoughtheliteraturehassuggestedthatROMAmodelisaneffectivetooltosupportsys-tematicadoptionof learninganalytics inHEIs[10,17], therehas been limited work that purposively involved differentstakeholdergroupstovalidatethefeasibilityofthistoolforLApolicydevelopment.Thecontributionofourworkistobridgethisgap,andextendtheuseoftheROMAmodeltoaddresschal-lenges recognised in the literature and raised by differentstakeholdergroups.WhilethefinalproductoftheSHEILApolicyframeworkwill
reflecttheperspectivesofvariousstakeholders,includinginsti-tutionalleadersanddecisionmakers,teachingstaff,students,andLAexperts,thispaperwillfocusonthefirstSHEILApolicyframework,whichwasdevelopedbasedon64interviewswithsenior managers from 51 European HEIs. Considering thescopeofthepaper,wewillpresentthreerepresentativecasestoillustratetheconceptoftheframework,aswellaspotentialwaystouseitforinstitutionalstrategicplanningandpolicyfor-mationforLA.
2 LITERATUREREVIEWIn spiteof thepotential toprovidebetter informationaboutstudent learningbehaviourandprogress, thereby improvingthequalityofeducationalofferingsandoptimisinglearning,LAhasmetanumberofchallengesthatimpedeitsadoptionataninstitutionallevel.Themostfrequentlyidentifiedissuesare(1)thedemandonresources,(2)issuesofethicsandprivacy,and(3)stakeholderengagementandbuy-in.Thesechallengesneed
to be tackled through strategic planningand a sound policyframework.Inthissection,weoutlineissuesidentifiedintheliterature under the three themes and introduce the ROMA(RAPID Outcome Mapping Approach) model, on which theSHEILApolicyframeworkisbased.
2.1 LearningAnalyticsChallenges2.2.1 DemandonResources.Thefirstmainissuecoverschal-lengesassociatedwithdataandtechnologicalinfrastructure,fi-nancialresources,andhumanresources.TheimplementationofLAtypicallyinvolvescomplexcomputingandaggregatingoflargeamountsofdata,inadditiontomanagementchallenges,suchastheintegrationofresearchtoolsintoexistinglearningenvironments[13].Thesetaskscanbedifficulttoperformwithtraditionaldatamanagementtechnologies[14].Asurveycar-riedoutbyEDUCAUSEtoinvestigateanalyticslandscapesinUShighereducationrevealedthatdata-qualityconcernsandsys-tem-integrationdifficultieswerepartofthemajorchallengestoembeddingtheuseofLAintoinstitutions[3].Thesefindingssuggestthatthere isaneedfora financial investment inad-vancing institutional data infrastructure to enable LA. How-ever,thesamestudybyEDUCAUSEalsofoundthatLAremainsaninterestratherthanamajorpriorityatmostinstitutions[3].Thisfindinghighlightsthechallengeofobtainingsufficientfi-nancialsupporttodevelopatechnologicalenvironmentforLAor appointinganalytics specialists inmanyHEIs if LAhas tocompetewithother institutional priorities. For example, an-otherEDUCAUSEreportbasedonthesamesurveydatapointedout that institutional analytics was twice as likely to be de-scribedasamajorpriorityaswaslearninganalytics,and4in10institutionsreportedlittleornoinvestmentinlearningan-alytics[32].Anotherkeydimensionishumanresources,whichincludes
boththeavailabilityofstafftimeandexpertisethatisrequiredtoimplementLA.Inacomplexeducationalsystem,theintro-ductionofasubtlechangecanmeetsubstantialresistancebe-causeoftheperceivedincreaseinworkloadforstaff[17].AsLAmakesuseofdatafromvarioussources,institutionsnotonlyneeddataexpertstoobtainandanalysegoodqualitydata,buttheyalso need theusers (e.g., administrators, teaching staff,andstudents)tohavebasicdatainterpretationskillsandtheabilitytoreflectondatacritically,inorderthatLAmayhavepositiveimpactoninformingdecisionsandchangingbehaviour[2,19,31].Thishasbeenidentifiedasacommongapbetweenneedsandsolutionsininstitutionalanalyticscapacity[18,25].
2.2.2 IssuesofEthicsandPrivacy.The secondmain issuehas been identified as amajor obstacle to gain buy-in fromstakeholders, especiallywhen the collection and use of dataseemtoriskintrudingprivacy[23,27].LikeallBigDataappli-cations,LAreliesonconstantandubiquitouscollectionofdatafromstudents.Thewiderangeandtypesofdatacollectedcouldinducediscomfortamongdatasubjectsduetoasenseofsur-veillance,leadingtoresistancetoLA[19].Moreover,whilean-
SHEILApolicyframework:informinginstitutionalstrategiesandpolicyprocessesoflearninganalytics LAK’18, March 2018, Sydney, Australia
3
onymity policies are commonly enforced inHEIs when per-sonaldataisused,itcanbedifficulttodelivercustomisedin-terventions without retaining a certain degree of individuallinkages [24]. Similarly,Greller andDrachsler acknowledgedthedilemmabetweenkeepingdataanonymousandexploitingthemostvalueofdata[12].Theyalsoarguedthatfearinducedbyethicsandprivacyissuescaneasilyleadtomisunderstand-ingsanddistrustininstitutions[8].Anotherkeyissueassociatedwithethicsandprivacyisin-
formedconsent [26].Rubeland Jonesquestion theextent towhich students canmake informedconsent [24].Theypointout thateducational institutionsmaybe transparent in theirdatapractices,butthecomplexityofalgorithmsstillmakesan-alytics a ‘black box’ formany.Moreover, the inherent infor-mationasymmetriesbetweendatacollectorsanddatasubjectsmeanstudentstendtohavelimitedknowledgeaboutwhocanaccesstheirdata,whattheydowiththedata,andwhatconse-quences intrusionsofprivacymaybe [8]. Similarly,PrinslooandSladeareconcernedaboutthebesttimetoseekconsentfromstudents.Theysuggestthatconsentseekingshouldfocusondownstreamusersratherthanonthetimeoftheinitialcol-lectionofdata,becausethebenefitsofopting-inoroutmaynotbeapparentat themomentwhenaLA service is introduced[22].Theconflictsbetweenmaximisingtheefficiencyandeffi-cacyofLAandrespectingdatasubjects’rightstocontroltheirown data can be challenging to institution adopting LA at alargescale.
2.2.3 StakeholderEngagementandBuy-in.Thethirdmainissue has been highlighted in a systematic literature reviewwhereTsaiandGaševićpointedoutthatHEIsstruggletofindcommongroundsamongdifferentstakeholdersregardingtheadoptionofLA,duetodiscrepanciesinexistingexperienceandknowledgeofdata,thereforeresultingindifferentunderstand-ingofpossiblebenefitsandoutcomesofLA[29].Moreover,ac-cordingtoTsaiandGašević,onlyahandfulofstudieshavetriedtoexplorestudentperspectivesregardingtheuseoftheirdataforlearninganalyticsortheimpactontheirlearningjourneys,despitethefactthatLAchampionsforalearningenvironmentthatislearner-centredandlearner-concerned[11].Thediffer-encesinperceptionsofLAamongstakeholderscanleadtoun-equalbuy-iniftheirneedsarenotmet,furtherresultingindis-trustinLAifconcernsarenotaddressed.Forexample,PrinslooandSladespecificallycalledforresearcherstoexplorepoten-tialconflictsbetweenstudents’concernswiththeirrighttoopt-outandtheimplicationsofpersonal-levelinterventionsfromHEIs[21].Adirectimpactofunequalengagementwithteachingpro-
fessionals is theweakpedagogicalgroundingofLAtechnolo-giesandimplementationdesign.Forexample,AliandotherspointedoutthatLA tools still needed tomovefromspottingstudents at risk toprovidingpedagogically informedsugges-tions[1],andMacfadyenandDawsonsuggestedthat institu-
tionsshouldbalancesolvingtechnicalchallengesanddevelop-ingpedagogicalplans[16].Similarly,FergusonandcolleagueshighlightedthatmuchworkonLAhasconcentratedonthesup-plyside,andconsiderablylessonthedemandside,forexampleconnectingLAwitheducationinwaysthatcantrulysupporttheeverydaylearning,teachingandassessmentwork[9].Fail-ingtoconsiderthepedagogicalcontextinwhichdataisgener-atedandinterpretedwillaffectteachingstaff’sperceptionsofthe usefulness of LA, thereby impeding broader buy-in andscalableactionsofLA[25].Thephenomenonofunequalengagementwithstakeholders
isalsoreflectedbytheabsenceofclearleadershiptodefinedi-rectionsforLAadoptionamongmanyHEIs[13],whichiscon-sideredakeyfactorassociatedwiththematurityofLAprac-ticesatan institutional level[6,18,25]. Inparticular, thein-volvementofinstitutionalleadersiscrucialtothedevelopmentofstrategiesandpoliciesforLA,whichcouldhelpmitigatethechallengesidentifiedsofar.Asnewpracticesinacomplexedu-cational system potentially disrupt traditional managementandorganisationalstructures,andthereforelikelytomeetre-sistance [17], it has been suggested that institutions shouldstartLAimplementationbydefiningastrategicplan[2,7,10].Moreover,studieshaveidentifiedthatexistingpoliciesrelatedtotechnicalstandardsforinteroperabilitydonotfullyapplytoLApractices[9],andtailoredLApoliciesforindividualinstitu-tionswillbeneededinordertoproperlyconsiderindividualinstitutionalcontextsineveryphaseofadoption[29].Withoutdedicatedinputfromhigh-leveldecisionmakers[7],itcanbedifficulttopressforthedevelopmentofLAspecificstrategiesandpoliciesthatmeettheneedsofindividualinstitutionsandthememberstherein.Inresponsetotheneedforastrategicframeworkandpolicy
toadoptLAsystematically,theSHEILAprojectusedtheRAPIDOutcomeMappingApproach(ROMA)toproduceapolicyde-velopment framework. The ROMAmodel was adopted as afoundation for thedevelopmentof theSHEILApolicy frame-workduetotheoriginalpurposeofROMAtosupportevidence-basedpolicydevelopmentandchangethroughactiveengage-mentwithrelevantstakeholders.ThemodelhasalreadybeensuggestedforsystemicadoptionofLAinHEIs[10,17].Thefol-lowingsubsectionintroducestheconceptoftheROMAmodel.
2.2.TheROMAModelinLearningAnalyticsCon-texts
TheROMAmodelwasdesignedbytheODI(OverseasDevelop-mentInstitute)toinformpolicyprocessesinthefieldofinter-nationaldevelopmentusingresearchevidence[33].Themodelbeginsbydefininganoverarchingpolicyobjective,whichisfol-lowedbysixstepsdesignedtoprovidepolicymakerswithcon-text-basedinformation:1)mappoliticalcontext,2)identifykeystakeholders,3)identifydesiredbehaviourchanges,4)developengagement strategy, 5) analyse internal capacity to effectchange,and6)establishmonitoringandlearningframeworks.
LAK’18, March 2018, Sydney, Australia Y.-S.Tsaietal.
4
Unlike traditional linear tools and approaches, ROMA is de-signed tobeused iteratively to informstrategic choicesandmeetunexpectedchanges(orchallenges)inacomplexsetting.Thismodelhasbeenadaptedtoguidetheplanningandimple-mentationofLAataninstitutionallevel[10,17](Figure1).
Figure1:TheRAPIDOutcomeMappingApproach[10]
Fergusonand colleaguesprovided twocase studiesofLApracticefromtheUKandAustraliatodemonstratehowtheo-reticalframeworkscouldbeoperatedintherealworldand,inparticular,howROMAcouldbeusedfortheplanningandim-plementationofLAinhighereducationcontextstomaximisethesuccessandimpactofLA.OurworkbuildsontheapproachadoptedbyFergusonandothers[10]tomapoutthestateofLAadoptionamongHEIsinEuropeusingROMA,andfurtherpro-videssuggestionstoguidepolicydevelopment.ThefollowingsectionaccountsfortheabovementionedmethodsadoptedtodeveloptheSHEILApolicyframework,followedbythreecasestudiesthathavecontributedtothispolicyframework.
3 METHODOLOGYTheSHEILApolicyframeworkwillbebasedonevidencefromawiderangeofdataincludinganinstitutionalsurveyadminis-teredtouniversitiesinEuropetounderstandthestateofadop-tion of LA (n=46), a Group Concept Mapping activity thatsoughtopinionsfromLAexpertsonessentialfeaturesofaLApolicy(n=30),64 institutional interviewswithmostly seniormanagers(e.g.,provosts,rector,deans,principals,viceprinci-pals,andvice/pro-vicechancellor)from51highereducationinstitutionsacross16countriesinEurope,andlocalconsulta-tionswithteachingstaffandstudentsatfourEuropeanhighereducationinstitutionsusingasurveymethodandafocusgroupmethod. The SHEILA policy frameworkwill be developed inphasesbasedonthefindingsfromtheabovementioneddata.Thispaperwillfocusontheoutputofthefirstphasedevel-
opment. The first SHEILA policy framework was developedbasedontheresultsofananalysisof64institutionalinterviewsthattookplacebetweenAugust2016andFebruary2017.Eachoftheseinterviewslastedfor30to60minutes.Thenumberofparticipants ineach interviewrangedfrom1to3, and someparticipantsfromthesameinstitutionattendedtheinterviewsseparately.Thisresultedinatotalnumberof78participants
from51institutions.Teninterviewquestionsweredevelopedtoinvestigate1)institutionalplansforLA,2)motivationsforLA,3)adoptedstrategy,4)strategydevelopmentprocesses,5)readinesspreparations,6)successandevaluation,7)successenablers,8)challenges,9)ethicalandprivacyconsiderations,and10)theinterviewee’sviewsofessentialelementsinaLApolicy.WeusedtheROMAmodelasatooltoanalyseeachinstitu-
tionalcasebymappingouttheirLArelatedactivitiesandchal-lengestothesixkeydimensionsofROMA.Duringthisprocess,we identifieda strong connectionbetween the sixROMAdi-mensions.Thatis,thesamechallengemaybeidentifiedinmul-tipledimensions,andanactionmaybeinformedbyconsidera-tionofmultipledimensionsatthesametime.WhiletheROMAmodelshouldbeappliediteratively,theredoesnotseemtobea definite order between the dimensions. Therefore, we de-cidedtotreatthemas ‘dimensions’ratherthan ‘steps’as ini-tially suggested by Young and Mendizabal [33], so as toacknowledgethefluiditybetweenthesixdimensions.Wesynthesisedthemappingresultsofthe51casesandcre-
atedacomprehensivetableofallactionsandchallengesidenti-fiedintheinterviews.Thisprocessresultedinalistof42actionpoints and 59 challenges across the six ROMA dimensions.Basedonthisresultandtheinterviewees’responsestoQues-tion10,wegenerated47policyquestionstoaddressthekeyactions and challenges. Thus, the SHEILA policy frameworkconsistsofacomprehensive listofadoptionactions,relevantchallengesandpolicyprompts,framedinthesixROMAdimen-sions. Figure 2 explains the concept and structure of theSHEILApolicyframework.
Figure2:TheSHEILApolicyframeworkstructure
Wegroupedtheactionpoints,challenges,andpolicyques-tionsbycommonthemesincludingcapabilities,culture,ethics&privacy,evaluation,financial&humanresources,infrastruc-ture,internal&externalsupport,management,methodology,purpose,andstakeholderengagement.Thesethemeshelpedus
SHEILApolicyframework:informinginstitutionalstrategiesandpolicyprocessesoflearninganalytics LAK’18, March 2018, Sydney, Australia
5
to identifythemainfocusofactionineachROMAdimensionandprevalentissuestoaddress.Thefollowingsectionsdiscussthemappingresultsofthree
casesthataredifferentfromeachotherbyinstitutionalsize,lo-cation,goals,andapproachestoLA.Whilethedatapresentedbelowonlymakesuppartofourpolicyframework,ourinten-tionistousethemtoillustratethedevelopmentprocessoftheSHEILApolicyframework,andtodemonstratehowtheSHEILApolicyframeworkcouldbeusedtoguidethedevelopmentofinstitutionalpoliciesandstrategicplanningforLA.
4 RESULTSInthissection,wepresenttheactionpointsundertakenbythethreeselectedinstitutionsandthechallengesthattheyfaced,followedbyalistofquestionstoreflectonwhendevelopingaLApolicyinsimilarcontexts.Eachofthestatementsisassoci-atedwithatheme.Section4.1presentstheprofilesofthethreecases, includingtheir approaches toLA. Section4.2presentsthemappingresultsofthethreecasesusingtheROMAmodel.
4.1 ThreecasesInstitutionAisbasedintheUKandhasmorethan30,000stu-dentsenrolled.Atthetimeoftheinterview,institutionAhadonecentraluniversitysponsoredLAprojectandanumberofsmallprojectsinitiatedbyindividualteachingstaff.Intermsoftheinstitutionaluptake,institutionAtookanexperimentalap-proachtoLA.Thatis,LAwasadoptednotasatooltosolveiden-tifiedproblems,butasatooltoexplorenewpossibilitiesandinnovations to enhanceexisting practice. Institution A’s goalwastouseLAtoenhancecurriculumdesignandstudentexpe-rience.InstitutionBisbasedinEstoniaandhasmorethan10,000
students enrolled.This institutionhada fewcourse-levelLAprojectspreviously,andwaspreparinganinstitutionalLApro-jectatthetimeoftheinterview.InstitutionBtookaproblem-basedapproachtoLA,whichisperceivedasapotentialsolu-tiontodealwithstudentdropouts.Thegoalwastounderstandstudents’ learning progress and provide interventions whenneeded.InstitutionCisbasedinSpainandhasmorethan30,000stu-
dentsenrolled.Atthetimeoftheinterview,institutionCdidnothaveanyinstitutionalLAproject,althoughthereweresmall-scaleprojectscarriedoutbyindividualresearchers.Themaingoaloftheseprojectswastoexploredatacollectedfromcur-rentandpastcoursestoidentifyopportunitiesforteachingin-novations.
4.2 SixROMAdimensions
Ananalysisof thethree casesusing theROMAmodel showsthatthemostcommonthemesofchallenges identified inDi-mension2(stakeholders)areethicsandprivacyrelatedissues,whilethoseinDimension3(desiredchanges),4(engagementstrategy),and6(monitoringframework)aremethodologyre-lated.Dimension5(capacityforchanges)examinedtheinter-nalcapacityoftheinstitutions,resultinginalongerlistofchal-lengesbeingidentifiedcomparedtotheotherdimensions.Thecommonchallengesinthisdimensionarerelatedtoculture,ca-pability,andinfrastructure.Incontrast,themappingofDimen-sion 1 (political context) did not identified shared themesamongthecomparativelyshorterlistofchallenges.Thefollow-ing subsectionsareorganisedaccording to the sixROMAdi-mensions.EachsectionbeginswithacriticalreflectiononthestateofadoptionofLAamongthethreecases,followedbythreetablesprovidingfurtherinformationoncorrespondingactions,challenges,andpolicypromptsrespectively.ThesetablesalsopresentaselectivepartoftheSHEILApolicyframework,asil-lustratedinFigure2.
4.2.1 Dimension 1 – Map political context ThemappingofDimension1revealedinstitutionaldriversandneedsforLA.BothCaseAandBfacedexternalpressuretoper-formquality evaluation,whichusually formspartof thekeyperformanceindicators(KPI)inHEIs(Table1).Therefore,itisparticularly important for these institutionsto reflecton thereasonsforadoptingLA–whetheritisforthebenefitsoftheinstitutionorforlearnersandteachers(Table3).WhileLAac-tivitiesinCaseCwerestillatagrass-rootlevel,thesamepolicyquestionswouldbeusefultoreflectonwhenplanningastrate-gicmovementtowardsinstitution-leveladoption.Thatis,alignindividual-level research activitieswith thewider universitystrategy,soastogainsupportfromseniormanagers/decisionmakers.TheneedtogainsupportfromkeyleadershiptoenablesystematicadoptionofLAhasalsobeenconfirmedbytheiden-tifiedchallenges(Table2)
Table1:Mappoliticalcontexts-actions
Case Action ThemeA Theinternaldriverwastousedatatoinform
teachingandlearningrelateddecisions,andanexternaldriverwastoprovidedataforaudits(e.g.NationalStudentSurvey).
Pur-pose
Giventhesizeoftheuniversity,itwasdecidedthatapilotstudywasneededtofindthebestwaytoextractandintegratedata.
Meth-odology
B The internal driver was to increase teachingqualityandlearningmotivations.Theexternaldriverwastoprovidedataforstate-levelqual-ity evaluations, which had previously high-lightedtheproblemofstudentdropouts.
Pur-pose
LAK’18, March 2018, Sydney, Australia Y.-S.Tsaietal.
6
C Akeydriverwastogainbetterunderstandingofcourserelatedactivitiessoastoimprovethecurriculumdesign.
Pur-pose
Table2:Mappoliticalcontexts-challenges
Case Challenges ThemeA Nochallengeswereidentified. N/AB Thereisnocentralguidancefromthegovern-
mentregardingtheuseofstudentdatainuni-versityfeedbacksystems.
Man-age-ment
C Decentralised leadership made it difficult totakeacentralisedapproachtoLA.
Meth-odology
Table3:Mappoliticalcontexts-policyprompts
Policy–questionstoreflecton ThemeWhatarethereasonsforintroducingLAtostudentsandstaff?Howdoinstitutionalobjectivesalignwithpersonalbenefitsforteachingstaffandstudents?
Pur-pose
4.2.2 Dimension 2 – Identify key stakeholders ThemappingofDimension2showedthattheadoptionofLAinthethreecasesinvolvedawiderangeofstakeholders,bothin-ternallyandexternally(Table4).Akeyimplicationforpolicyistoconsidertheresponsibilitiesandrightsofeveryoneinvolved,inadditiontotheimpactonthem(Table6).CaseB,inparticu-lar,facedanethicaldilemmaabouthowtomakeopt-outop-tionsavailablewhile addressing institutional challenges thatinvolveeverymemberoftheinstitution(Table5).Whilethereis no easy solution for this challenge, defining the circum-stancesofenforcingopt-out/-inoptions,anonymity,andlim-itedaccesstodatainapolicycaneffectivelyminimiseconflicts.Incontrast,CaseCwasconcernedaboutdatare-identification,whichwouldneedtobeaddressedbyevaluationactioninDi-mension6(seeSection4.2.6).Animplicationofthischallengeforpolicyistodefinerulesaboutsharingdatawithresearchersandexternalparties.
Table4:Identifykeystakeholders-actions
Case Action ThemeA The primary internal stakeholders included
students,teachingstaff,seniormanagersandaworkinggroupmadeofrepresentatives fromvariousunits.TheexternalstakeholderwasaLAserviceproviderthatofferedawarehouseandanalyticsexpertise.
Stake-holderengage-ment
B The primary internal stakeholders includedstudents,teachingstaff,ITofficers,seniorman-agers,andthedepartmentofacademicstudies.
Stake-holder
The need to involve external stakeholders,such as LA experts and data scientists, wasidentified.
engage-ment
C Themain stakeholderswereresearchersandIT officers. However, there was indirect en-gagement with external researchers throughthe engagement of LA literature and confer-ences.
Stake-holderengage-ment
Table5:Identifykeystakeholders-challenges
Case Challenges ThemeA It was difficult to define ownership and re-
sponsibilities among professional groupswithintheuniversity.
Man-age-ment
B Theprovisionofopt-outoptionsconflictswiththegoaltotackleinstitutionalchallengesthatinvolveallinstitutionalmembers.
Ethics&Privacy
C Anonymiseddatacouldpotentiallybere-iden-tifiedwhenmatchedwithotherpiecesofdata.
Ethics&Privacy
Table6:Identifykeystakeholders-policyprompts
Policy–questionstoreflecton ThemeWhoisthepolicyfor?Howwillresponsibilitiesbedefinedforeachstake-holder?
Stake-holderengage-ment
Whosedatawillbecollected? Meth-odology
Howwillconsentbeobtained?Isthereanoptiontoopt-outof(oroptinto)anydatacollectionandanalysis?Whocanaccessthedata?Howwill anonymitypoliciesbeappliedto thepro-cessingandpresentationofdata?Willdatabesharedwithresearchers?Willdatabesharedwithexternalparties?Isitjustifi-able?
Dataman-age-ment
4.2.3 Dimension 3 – Identify desired behaviour changes ThemappingofDimension3showedthattheexpectedchangesforCaseBwereparticularly ‘institution-focused’,whilethoseidentifiedinCaseCwereteacher-focused(Table7).AlthoughCaseAexpectedtoseebehaviourchangesamongallthreelev-elsofstakeholders,therewasaconcernthatexpectationsmaynotbemet(Table8).Asimilarconcernaboutreturnsoninvest-mentwasobservedinCaseBwhereLAwasalsodrivencen-trallybytheinstitution.Therefore,itisimportantthatthepol-icynotonlyguidesdecisionmakersto focusonchangesthatmeaningfullyreflectthegoalssetoutforLA(Table9),butalsoarangeofindicatorsthatcantrulyreflectthesechanges ina
SHEILApolicyframework:informinginstitutionalstrategiesandpolicyprocessesoflearninganalytics LAK’18, March 2018, Sydney, Australia
7
specificinstitution’scontext.Thelattercouldbedefinedassuc-cessindicators,assuggestedlaterinDimension6(seeSection4.2.6).
Table7:Identifydesiredbehaviourchanges-actions
Case Action ThemeA Academic staff will better understand stu-
dents’ learning problemsand offer supportaccordingly.Studentswillbeabletoreflectonhowtheylearn,andmakelearningplansaccordingly.Theinstitutionwillbeabletomakebetterde-cisions to support learning and teachingbasedonanoverviewoflearningandteach-ingeffectiveness.
Purpose
B Studentdropoutrateswilldecrease.Students will be provided with regular re-portsabouttheirlearningprogress.Theinstitutionwillmakebetterdecisionstoenhanceteachingqualityandkeepstudentsmotivated.
Purpose
C Academic staff will better understand stu-dent learning behavior, thereby improvingthewaytheyteach.The institution will improve the quality oftheireducationalservices.
Purpose
Table8:Identifydesiredbehaviourchanges-challenges
Case Challenges ThemeA Anexperimentalapproachissusceptibletoa
senseofuncertaintyaboutthereturnonin-vestment.
Method-ology
B It is unclear if a problem-based approachguaranteesasolution.
Method-ology
C Nochallengeswereidentified.
Table9:Identifydesiredbehaviourchanges-policyprompts
Policy–questionstoreflecton ThemeWhatchangeswillLAbringtothecurrentsituation?Whyarethesechangesimportanttous?
Purpose
Whowillbenefitfromlearninganalytics?Howwillthepurposeoflearninganalyticsbecom-municatedtoprimaryusers?
Stake-holderengage-ment
4.2.4 Dimension 4 – Develop engagement strategy
Themapping of Dimension 4 showed that engagement datawasconsideredprimarydataforLAinthethreecases(Table10).Theimplicationforpolicyistodefinetherangeofdatabe-ingcollectedandencourage‘meaningfulselection’ofdata,sothatLAwillnotbedrivenbydata,butbylearningorteachinggoals(Table12).Itisalsocrucialtoincludestudentsandteach-ersintheinterpretationofdatasoastocontextualisedataandincrease thevalidityof analytics.The challenges thatCaseAandCfocusedonsuggesttheimportanceofincludingthesekeystakeholdersineffortstoimprovetheefficacyofLA(Table11)Acommonstrategysharedbyallthreecasesistosetupawork-inggrouptodriveLA.Itisimportantthatthepolicystatestheresponsibilitiesoftheworkinggroup,particularlytheirroleinensuringthatLAwillbeusedresponsiblywithintheinstitution.Forexample, theworkinggroupatCaseBwillneedtomakesurethatrelevantdataprotectionregulationshavebeencon-sulted,asitisnotevidentinthereportedactions.
Table10:Developengagementstrategy-actions
Case Action ThemeA TheinitialengagementwithLAwasguided
byJisc’sCodeofPracticeforLearningAnalyt-ics.Therewerepreparationstodevelopaninsti-tutional policy to provide a framework fortheuseofLAinthelocalcontext.
Ethics &privacy
TwoLAspecialistsandaworkinggroupweresetup to facilitateapilotprojectwithaLAserviceprovider,engagewithresearchactiv-ities,anddevelopinstitutionalstrategies.
Humanre-sources
TheinitialpreparationsincludedareviewofexistingLAcases.Thesourcesofdatausedinthepilotprojectincludedinteractionsinvirtuallearningenvi-ronments, Student Record Systems, andcoursemarks.Sixty-fiveonlineMSccourseswereinvolved.
Method-ology
B AdiverseworkinggroupwassetuptodriveLAactivities.
Humanre-sources
Theworkinggroupwillinitiatecommunica-tionsamongdifferentstakeholders.
Stake-holderengage-ment
Theinitialpreparationsincludedareviewofexisting LA cases and visits to other Euro-pean universities to learn from best prac-tices.ThedatasourcesincludedengagementdatainLMS(LearningManagementSystem)and
Method-ology
LAK’18, March 2018, Sydney, Australia Y.-S.Tsaietal.
8
data held in SIS (Student Information Sys-tem).
C There were consultations on the SpanishLOPD(OrganicLawonProtectionofPersonalData).
Ethics &Privacy
TherewasaplantosetupaworkinggrouptopromoteLAamongteachingstaffandde-velopethicalguidelines.
Humanre-sources
Social interaction data was extracted fromdiscussionforumsintheLMS.
Method-ology
Table11:Developengagementstrategy-challenges
Case Challenges ThemeA Overrelyondataandfailtoconsidertheex-
perienceandknowledgeofinstructor/tutorsaboutstudents.
Method-ology
B While there was funding support from thegovernmenttodevelopstudentfeedbacksys-temsamongEstonianuniversities,therewasnostate-levelcoordinationtoinitiatecollab-oration among universities that have re-ceivedthegrant.
Manage-ment
C Focus on identifying students at risk andoverlook thepedagogicaldesignof curricu-lumorlearningsupport
Method-ology
Table12:Developengagementstrategy-policyprompts
Policy–questionstoreflecton ThemeWhataretheobjectivesforLA? PurposeWhatkindsofdatawillbecollectedtoachievetheseobjectives?Whatisthescopeofdatacollection?How will the results of analytics be interpretedwithinthecontext?Willteachingstafforstudentsbeinvolvedintheprocess?Whowilloverseeethicalconductsrelatedtolearn-inganalytics?
Method-ology
4.2.5 Dimension 5 – Analyse internal capacity to effect change ThemappingofDimension5showedthattheevaluationofin-ternalcapacityfocusedonfinancial,infrastructure,andhumancapacity(Table13).Acommonchallengesharedbythethreecases was in gaining wide support from the teaching staffamong whom analytical literacy and time availability weremainissuestodealwith(Table14).Theimplicationforpolicyis to ensure the availability of communication channels andsupport resources among different stakeholders (Table 15).While all cases identified the challenge of accessing certain‘useful’ data, Cases A and B recognised that ethical conduct
needsanenablinginfrastructure.Thus,itiscrucialthatthepol-icyprovidesguidelinestokeeptheinfrastructureupdatedwithregardtocurrentdataprotectionrequirements.
Table13:Analyseinternalcapacitytoeffectchange-ac-tions
Case Action ThemeA Ariskevaluationwasperformedtoanalyse
internalcapacity.Method-ology
B Therewasgovernmentfundingforthedevel-opmentoffeedbacksystemstosupportstu-dents.
Financialresources
C Therewas an evaluation of the availabilityandusefulnessofdatafromtheLMS.Interest was expressed in cross-institutioncollaborationonLAresearchprojectstoen-hancetheintegrationofLA.
Infra-structure
Table14:Analyseinternalcapacitytoeffectchange-chal-lenges
Case Challenges ThemeA 2018GDPR(EuropeanGeneralDataProtec-
tion Regulation) will bring changes to thewaytheuniversitydealtwithstudentdata.
Method-ology
The existing data infrastructure could notdealwithindividualopt-outs.Therewasno singlepermissiontouse stu-dentdataacrosstheinstitution.Some useful data remains inaccessible, e.g.the usage record of the digital library waskeptbypublishers.
Infra-structure
IfInstitutionAfailedtomanageonestudent’srequesttobeexcludedproperly,theunhap-pinessofonestudentmightspreadtoothersandstartaninstitution-wideobjection.Thebuy-infromteachingstaffwaspolarised.
Culture
B Thecultureofusingdatatoinformdecision-makingwasimmature.Although compulsory trainingwas plannedfor teaching and support staff, it was notclearhowtofosterownershipofLAamongstaff.ThebenefitofusingLAtosupportdecision-makingwascleartoseniormanagersbutnottoteachingstaff.
Culture
The existing infrastructure is not matureenough toprocessdata from theLMSortocopewithprivacyrequirements,suchasal-lowingindividualopt-outs.
Infra-structure
SHEILApolicyframework:informinginstitutionalstrategiesandpolicyprocessesoflearninganalytics LAK’18, March 2018, Sydney, Australia
9
Data that ispotentiallyuseful for achievingthegoalsofLAmaynotbeaccessibleduetoprivacyissues.There was a skills gap in analytics and LAproject design, which posed questions re-gardingthevalidityofthecurrentapproachtoLA.
Capabili-ties
C Theskillsrequiredtounderstandandinter-pret visualised data needed to be installedamongteachingstaff.
Capabili-ties
Worriesaboutthetimedemandsinincorpo-ratingLAintoteachingoutweighedtheper-ceivedbenefitsofLA,andreducedthemoti-vationtoattendrelevanttraining.
Culture
CertaindataoutsidetheLMSishard toac-quire,suchassocialinteractionsinaphysicalclassroom.
Infra-structure
Table15:Analyseinternalcapacitytoeffectchange-pol-icyprompts
Policy–questionstoreflecton ThemeHowwilldataintegritybeachieved? Method-
ologyHowwillthedatabestoredanddisposed?Howoftenwilltheefficiencyandsecurityofexist-ingdatainfrastructurebeevaluated?
Datamanage-ment
Arethererelatedpolicies inthe institutional/na-tional/ international level that the LA policy sitsalongside/above/below?
Policymanage-ment
Whatcommunicationchannelsorfeedbackmecha-nismswillbeinplace?Whattrainingwillbedeployed?Willitbecompul-sory?
Stake-holderengage-ment
4.2.6 Dimension 6 – Establish monitoring and learning frame-works ThemappingofDimension6showedthatnoneofthethree
institutionshaddevelopedsuccesscriteriaordefinedmonitor-ingprocedures,perhapsdue to theearly stagesof adoption.However,thechallengesthatconfrontedthemindicatetheur-gency and importance to define success measures for LA intheircontexts,particularlywiththegroundingoflearningandteaching theories (Table 16). More importantly, the policyneedstoraiseawarenessaboutinadvertentconsequencesthatmayresult fromanalytics,andsuggestproceduretomonitoranddealwiththeserisks(Table17).
Table16:Establishmonitoringandlearningframeworks-challenges
Case Challenges Theme
A Therewasafearof failingtomeetexpecta-tions,resultinginabadnameforLA.
Method-ology
B It remains questionable whether studentdropoutrateisthebestsuccessindicatorfortheinstitutionalLAproject.
Method-ology
C Thecaptureddataoftimespentonlinemaynottrulyreflectlearning.The design and implementation of LAmayfailtoconsiderpedagogicaltheories.
Method-ology
Table17:Establishmonitoringandlearningframeworks-policyprompts
Policy–questionstoreflecton ThemeHowwillsuccessbemeasured?Whataresuccessindicators?Whatarethemechanismsthatdealwithinadvert-entconsequences?Whowillcarryouttheevaluationofimpact?
Evalua-tion
Howoftenwillthepolicybereviewedandupdated?Whowillberesponsibleforthepolicy?
Policymanage-ment
5 DISCUSSIONTheassociatedthemesthathaveemergedinthemappingre-sultsshowadifferentfocusforeachROMAdimension.Dimen-sion1 (mappingpolitical context) focuseson identifying the‘purpose’foradoptingLAinaspecificcontextsoastodriveac-tionsintheotherdimensions.Dimension2(identifykeystake-holders)isdrivenbytherecognitionthattheimplementationofLAinasocialenvironment involvescollectiveeffortsfromdifferent stakeholders. Dimension 3 (identify desiredbehav-iourchanges) setsobjectives,which reflectback to the ‘pur-pose’ofadoptingLA.Dimension4(developengagementstrat-egy)definesapproachestoachievingtheobjectivesbyaddress-ingaspectsthatcouldotherwisebecomechallenges,asidenti-fied in the literature: resources, ethics&privacy, and stake-holderengagementandbuy-in(seeSection2.1).Dimension5(analyse internal capacity to effect change) focuses on as-sessing the availability of existing resources (e.g., data andfunding)andidentifyingchallenges(risks).Dimension6(es-tablishmonitoringandlearningframeworks)iscurrentlyab-sentinallthreecases.ThismappingprocessillustrateshowtheROMAmodelcan
beusedtoexamineexistingLApracticesandrefinestrategies.Forexample,themappingresultsshowthatallthreecasesstillneedtoconsiderwhatitmeanstobesuccessfulwithLAandwhatsuccesslookslike(Dimension6),soastobetterinformactionsrelatedtootherdimensions.Theactionstakenbythethree cases also contributed to the action elements in theSHEILApolicy framework(Figure2),whichcouldbeusedtoinitiatestrategicplanningforearlyadopters.
LAK’18, March 2018, Sydney, Australia Y.-S.Tsaietal.
10
Intermsofchallengesthatconfrontedthethreecases,themappingofDimension5identifiedkeythemesaroundculture,capability,andinfrastructure.Thisresultcoincideswithtwoofthe threekeyLAchallenges identified in the literature–de-mandonresourcesandstakeholderengagementandbuy-inasintroducedinSection2.1.Asaresult,thepolicyquestionsfocusonmanagementissuesarounddataintegrityandsecurity,andchannels for stakeholdertrainingandcommunicationwithintheinstitution.Theotherkeychallenge–ethicsandprivacy–wasparticularlyhighlighted in themappingofDimension2.Thisreafxirmstheimportanceandurgencyofaddressingethicsandprivacy issues that couldotherwise impedebuy-in fromstakeholders.Tothisend,thepolicyquestionsparticularlyfo-cusonmanagement issuesaroundprivacy, suchas consent-seeking,dataaccess,anonymityprinciples,anddatasharing.Whileapolicydoesnotnecessarilyprovidedirectsolutions
totheidentixiedchallenges,thequestionsintheSHEILApolicyframeworkintendtopromptanswersthatcouldserveassuit-ablecodeofpracticetomitigatethechallenges.Forexample,answerstothepolicyquestion–“howwillanonymitypoliciesbeappliedtotheprocessingandpresentationofdata”(seeTa-ble6)maynotprovidesolutionstothedatare-identixicationchallengeidentixiedbyCaseC(seeTable5),asitmaynotbeforeseenbeforedifferentdatasetsareintegrated.However,apolicy could suggest that a reviewand testprocess for suchrisks be carried out by data specialists before data ismadeavailabletoawiderpopulationofstakeholders.Thismayfur-therinformactionsofDimension4and5,astheavailabilityofdatacouldbedeterminedbytheassociatedrisksofprivacyandconsequentlyaffectengagementstrategy.Asidentifiedintheliterature,stakeholderengagementand
buy-inhasadirectimpactonthescalabilityandsustainabilityofLA,whichneedtobesupportedbystrategicplanning,ledbyinstitutionalleaders,andinformedbypedagogicalknowledgepossessedbyteachingprofessionals.ThisissueisreflectedinthemappingresultsofchallengesassociatedwithDimension1,3and4,where‘methodology’and‘management’arekeyissues.Asaresult,thepolicyquestionsfocusondefiningthepurposeofimplementingLAandconsideringthevalueofLAtoallrele-vant stakeholders and the specific context of the institution.Basedontheidentifiedpurpose,themethodologyadoptedtoachievethechosengoalshouldalsobestatedinapolicy,assug-gestedinDimension4.
6 CONCLUSIONWehavepresented three institutions’ approaches toLAandchallengesthatconfrontedtheminthispaper.UsingtheROMAmodel,weanalysedactionscarriedoutbythese institutions.WeextendedandadaptedtheuseofROMAfurtherbyincludingchallengesunderthesixdimensions.Thereafter,wedevelopeda set of questions to beaddressedwhen formulating policy.Thismapping process demonstrated the evidence-based ap-proachthatweadoptedtodeveloptheSHEILApolicyframe-work,which contributes three typesof informationvaluable
forasystematicadoptionofLA–actions,challenges,andpolicy.Theframeworkcouldbeusedtoguidethedevelopmentofin-stitutionalpoliciesandstrategicplanningforlearninganalyt-ics,toevaluateinstitutionalreadinessforLAandtobenchmarkbestpractices.ThispaperhaspresentedaselectivepartofthefirstSHEILA
policyframeworkthroughthreechosencases.Thelistofpolicypromptspresentedin thispaperwere selected to reflect thethreeparticularcases.TheframeworkwasdevelopedbasedonaseriesofinterviewswithpredominantlyseniormanagersinHEIs.Therefore,itparticularlyreflectstheperspectivesofthisgroup of stakeholders. Our futurework aims to incorporatefindingsfromotheron-goingresearchactivities,whichexploreviewsfromotherkeystakeholderssuchasteachersandstu-dents,regardingtheadoptionofLA.ACKNOWLEDGMENTSThisworkwassupportedbytheErasmus+ProgrammeoftheEuropean Union [562080-EPP-1-2015-1-BE-EPPKA3-PI-FOR-WARD].TheEuropeanCommission’ssupport fortheproduc-tionofthispublicationdoesnotconstituteanendorsementofthecontents,whichreflectstheviewsonlyoftheauthors,andtheCommissionwillnotbeheldresponsibleforanyusewhichmaybemadeoftheinformationcontainedtherein.Theprojectinvolvedcollaborativeinputfromallthepartnersinvolvedandtheircontributionsarehighlyappreciated.Wewouldalsoliketogivethankstoourresearchparticipants fortheirvaluablecontributions.
REFERENCES
[1] Ali,L.,Asadi,M.,Gašević,D.,Jovanović,J.andHatala,M.2013.Factorsinflu-encingbeliefsforadoptionofalearninganalyticstool:Anempiricalstudy.Computers & Education. 62, (Mar. 2013), 130–148.DOI:https://doi.org/10.1016/j.compedu.2012.10.023.
[2] Arnold,K.E.,Lynch,G.,Huston,D.,Wong,L.,Jorn,L.andOlsen,C.W.2014.BuildingInstitutionalCapacitiesandCompetenciesforSystemicLearningAnalyticsInitiatives.ProceedingsoftheFourthInternationalConferenceonLearningAnalyticsAndKnowledge(NewYork,NY,USA,2014),257–260.
[3] Arroway,P.,Morgan,G.,O’Keefe,M.andYanosky,R.2016.LearningAnalyt-icsinHigherEducation.ECAR.
[4] Bichsel,J.2012.AnalyticsinHigherEducation:Benefits,Barriers,Progress,andRecommendations.ECAR.
[5] Code of practice for learning analytics: 2015.https://www.jisc.ac.uk/guides/code-of-practice-for-learning-analytics. Ac-cessed:2016-06-07.
[6] Colvin,C.,Dawson,S.,Wade,A.andGašević,D.2017.AddressingtheChal-lengesofInstitutionalAdoption.HandbookofLearningAnalytics.C.Lang,G.Siemens,A.Wise,andD.Gaśević,eds.SocietyforLearningAnalyticsRe-search.281–289.
[7] Colvin,C.,Rogers,T.,Wade,A.,Dawson,S.,Gasevic,D.,Shum,S.B.,Nelson,K.,Alexander,S.,Lockyer,L.,Kennedy,G.,Corrin,L.andFisher,J.2015.Stu-dentRetentionandLearningAnalytics:ASnapshotofAustralianPracticesandaFrameworkforAdvancement.TheAustralianGovernmentOfficeforLearningandTeaching.
[8] Drachsler,H.andGreller,W.2016.PrivacyandAnalytics:It’saDELICATEIssueaChecklist forTrustedLearningAnalytics.ProceedingsoftheSixthInternationalConferenceonLearningAnalytics&Knowledge (NewYork,NY,USA,2016),89–98.
[9] Ferguson,R.,Brasher,A.,Clow,D.,Cooper,A.,Hillaire,G.,Mittelmeier, J.,Rienties,B.,Ullmann,T.andVuorikari,R.2016.ResearchEvidenceontheUseofLearningAnalytics:ImplicationsforEducationPolicy.
[10] Ferguson,R.,Macfadyen,L.P.,Clow,D.,Tynan,B.,Alexander,S.andDawson,S.2014.SettingLearningAnalyticsinContext:OvercomingtheBarriersto
SHEILApolicyframework:informinginstitutionalstrategiesandpolicyprocessesoflearninganalytics LAK’18, March 2018, Sydney, Australia
11
Large-ScaleAdoption.JournalofLearningAnalytics.1,3(Sep.2014),120–144.
[11] Gašević,D.,Dawson,S.andSiemens,G.2015.Let’snotforget:Learningan-alytics are about learning. TechTrends. 59, 1 (Jan. 2015), 64–71.DOI:https://doi.org/10.1007/s11528-014-0822-x.
[12] Greller,W.andDrachsler,H.2012.TranslatingLearningintoNumbers:AGenericFrameworkforLearningAnalytics.EducationalTechnology&Soci-ety.15,3(2012),42–57.
[13] HigherEducationCommission2016.FromBrickstoClicks-ThePotentialofDataandAnalyticsinHigherEducation|HigherEducationCommission.
[14] Jeremic,Z.,Kumar,V.andGraf,S.2017.MORPH:Supportingtheintegrationoflearninganalyticsatinstitutionallevel.ProceedingsoftheSeventhInter-nationalLearningAnalytics&KnowledgeConference(NewYork,NY,USA,2017),596–597.
[15] Knight,D.B.,Brozina,C.,Stauffer,E.M.,Frisina,C.andAbel,T.D.2015.De-velopingaLearningAnalyticsDashboardforUndergraduateEngineeringUsingParticipatoryDesign.(Jun.2015),26.485.1-26.485.11.
[16] Macfadyen,L.andDawson,S.2012.Numbersarenotenough.Whye-learn-inganalyticsfailedtoinformaninstitutionalstrategicplan.FacultyofEdu-cation-Papers(Archive).(Jan.2012),149–163.
[17] Macfadyen,L.P.,Dawson,S.,Pardo,A.andGaševic,D.2014.EmbracingBigDatainComplexEducationalSystems:TheLearningAnalyticsImperativeandthePolicyChallenge.Research&PracticeinAssessment.9,(2014),17–28.
[18] Norris,D.M.andBaer,L.L.2013.BuildingOrganizationalCapacityforAna-lytics.
[19] Pardo,A.andSiemens,G.2014.Ethicalandprivacyprinciplesforlearninganalytics.BritishJournalofEducationalTechnology.45,3(May2014),438–450.DOI:https://doi.org/10.1111/bjet.12152.
[20] Park,Y.andJo,I.-H.2015.Developmentofthelearninganalyticsdashboardtosupportstudents’learningperformance.JournalofUniversalComputerScience.21,1(2015),110–133.
[21] Prinsloo,P.andSlade,S.2017.EthicsandLearningAnalytics:Chartingthe(Un)Charted.HandbookofLearningAnalytics.C.Lang,G.Siemens,A.Wise,andD.Gašević,eds.SocietyforLearningAnalyticsResearch.49–57.
[22] Prinsloo,P.andSlade,S.2015.StudentPrivacySelf-management:Implica-tionsforLearningAnalytics.ProceedingsoftheFifthInternationalConfer-enceonLearningAnalyticsAndKnowledge(NewYork,NY,USA,2015),83–92.
[23] Roberts,L.D.,Howell,J.A.,Seaman,K.andGibson,D.C.2016.Studentatti-tudestowardlearninganalyticsinhighereducation:“Thefitbitversionofthe learning world.” Frontiers in Psychology. 7, (Dec. 2016), 1–11.DOI:https://doi.org/10.3389/fpsyg.2016.01959.
[24] Rubel,A.andJones,K.M.L.2016.Studentprivacyinlearninganalytics:Aninformationethicsperspective.TheInformationSociety.32,2(Mar.2016),143–159.DOI:https://doi.org/10.1080/01972243.2016.1130502.
[25] Siemens, G., Dawson, S. and Lynch, G. 2013. Improving the Quality andProductivityoftheHigherEducationSector:PolicyandStrategyforSystems-levelDeploymentofLearningAnalytics.SocietyforLearningAnalyticsRe-searchfortheAustralianOfficeforLearningandTeaching.
[26] Slade,S.andPrinsloo,P.2013.LearningAnalytics:EthicalIssuesandDi-lemmas.American Behavioral Scientist. 57, 10 (Mar. 2013), 1510–1529.DOI:https://doi.org/10.1177/0002764213479366.
[27] Slade, S. andPrinsloo,P. 2014. Studentperspectiveson theuse of theirdata:Betweenintrusion,surveillanceandcare.ChallengesforResearchintoOpen&DistanceLearning:DoingThingsBetter–DoingBetterThings(Ox-ford,UK,Oct.2014),291–300.
[28] TheOpenUniversity2014.PolicyonEthicaluseofStudentDataforLearn-ingAnalytics.TheOpenUniversity.
[29] Tsai,Y.-S.andGašević,D.2017.Learninganalyticsinhighereducation—challengesandpolicies:Areviewofeightlearninganalyticspolicies.Pro-ceedingsoftheSeventhInternationalLearningAnalytics&KnowledgeCon-ference(NewYork,NY,USA,2017),233–242.
[30] Tsai,Y.-S.andGašević,D.2017.TheStateofLearningAnalyticsinEurope–ExecutiveSummary–SHEILA.
[31] Wolff,A.,Moore,J.,Zdrahal,Z.,Hlosta,M.andKuzilek,J.2016.DataLiteracyforLearningAnalytics.ProceedingsoftheSixthInternationalConferenceonLearningAnalytics&Knowledge(NewYork,NY,USA,2016),500–501.
[32] Yanosky,R.andArroway,P.2015.TheAnalyticsLandscapeinHigherEdu-cation.ECAR.
[33] Young,J.andMendizabal,E.2009.HelpingResearchersBecomePolicyEn-trepreneurs-HowtoDevelopEngagementStrategiesforEvidence-basedPol-icy-making.OverseasDevelopmentInstitute.