learning analytics in small-scale teacher-led innovations: … · 2017-02-27 · learning analytics...

(2016).Learninganalyticsinsmall-scaleteacher-ledinnovations:Ethicalanddataprivacyissues.JournalofLearningAnalytics,3(1),43–65.http://dx.doi.org/10.18608/jla.2016.31.4

ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 43

Learning Analytics in Small-Scale Teacher-Led Innovations: Ethical and Data Privacy Issues

MaríaJesúsRodríguez-Triana

REACT ÉcolePolytechniqueFédéraledeLausanne(EPFL)

Lausanne,Switzerland [email protected]

AlejandraMartínez-Monés

GSIC-EMIC Dpto.Informática,UniversidaddeValladolid

Valladolid,Spain

SaraVillagrá-Sobrino GSIC-EMIC

Dpto.Pedagogía,UniversidaddeValladolid Valladolid,Spain

ABSTRACT:Asafurthersteptowardsmaturity,thefieldoflearninganalytics(LA)isworkingonthedefinitionofframeworksthatstructurethelegalandethicalissuesthatscholarsandpractitioners must take into account when planning and applying LA solutions to theirlearning contexts. However, current efforts in this direction tend to be focused oninstitutionalhighereducationapproaches.Thispaper reflectson theneed toextend theseethicalframeworkstocoverotherapproachestoLA;moreconcretely,small-scaleclassroom-orientedapproaches thataimtosupport teachers in theirpractice.This reflection isbasedon three studies where we applied our teacher-led learning analytics approach in highereducationandprimaryschoolcontexts.Wedescribetheethicalissuesthatemergedintheselearning scenarios, and discuss them according to three dimensions: the overall learninganalyticsapproach,theparticularsolutiontolearninganalyticsadopted,andtheeducationalcontextswheretheanalyticsareapplied.Weseethiseffortasafirststeptowardsthewiderobjective of providing a more comprehensive and adapted ethical framework to learninganalytics that is able to address the needs of different learning analytics approaches andeducationalcontexts.Keywords:Learninganalytics,ethics,privacy,highereducation,primaryeducation

1 INTRODUCTION Theincreasingtrendtowardsmassivedatacollectionineducationalsettingshasraisednewethicalconcerns in the learning analytics (LA) research community. On the one hand, there is a need toidentifystudentsacrossplatformsandretrieveasmuchdataaspossibleinordertoobtaininformedanalysisaboutthelearningprocesses.Ontheotherhand,otheraspectsinfluencetheadoptionandacceptability of learning analytics approaches, such as data ownership and openness, potentialabuse and the need for new key competences to interpret and act on learning analytics results



(Greller&Drachsler,2012).Theseandotheraspectsarepartoftheethicalconcernsthatthefieldoflearninganalyticsmustfaceinordertogrowasamaturediscipline. Several authors have reflected on the ethical issues that affect the field (Slade& Prinsloo, 2013;Sclater,2014)andhavemadeproposalstodealwiththem,likethesetofdesignguidelinesproposedby Pardo and Siemens (2014). However, most of these analyses and proposals apply to highereducationinstitutionalcontexts.Littlereflectionhasbeendoneontheimplicationsofusinglearninganalyticsinsmaller-scalecontextswhereteachersusethedatatomanagetheiruniversityorschool-level classrooms,especiallyon the latter.However, aspointedoutbyGriffiths (2012), thekindofethical considerations thatmust be taken into account are different depending on the approachtakentolearninganalytics.Therefore,thereisaneedtoreflectonwhatethicalaspectsarerelevantin the applications of learning analytics to small-scale teaching practices, and especially in schoolcontexts. Duringrecentyears,wehaveworkedonthesupportofteachers’orchestration(Dillenbourg,2013)ofCSCLactivitiesinblendedlearningscenarios.Withtheaimofprovidingteacherswithmeaningfuland easy-to-appropriate data, we have proposed both scripting and monitoring processes, eachawareof theother, so that theanalysis and resultsprovidedby the learninganalytics systemareinfluencedbytheinformationprovided(bytheteacher)atdesigntime(Rodríguez-Triana,Martínez-Monés, Asensio-Pérez, & Dimitriadis, 2012; Rodríguez-Triana, Martínez-Monés, Asensio-Pérez, &Dimitriadis, 2013). Both processes are supported by a technological infrastructure — a dataintegration architecture— able to integrate different kinds of data sources, including LMS, tools,and user-generated data (Rodríguez-Triana, Martínez-Monés, & Asensio-Pérez, 2011). Thisintegrationaimstoapplylearninganalyticstotheexistingtechnologicalecosystemswithwhichtheusers(teachersandstudents)arefamiliar. These proposals were designed and validated following a design-based research (DBR) processapplied to seven cases in higher education (Rodríguez-Triana, Martínez-Monés, Asensio-Pérez, &Dimitriadis,2015).Totestthecapabilityofoursolutionstoadapttodifferenteducationallevels,weappliedthemtoaprimaryschoolcase.Wefoundthatwhilewecouldeasilymanagethetechnicalaspectsofdataretrieval,analysis,andvisualization,emergentissuesrelatedtodataownershipandcontrol,studentidentity,andotherlegalandethicalconcernsweremoredifficulttohandle. We present, in this paper, a reflection on how the ethical frameworks proposed in LA literatureapplytoourteacher-centredclassroom-basedapproachandtothetwoaforementionededucationalcontexts. The two studies presented in the paper illustratewith concrete exampleswhich ethicalissuesarerelevantinlearninganalyticswhenappliedtosmall-scaleteacher-ledinnovationinhighereducation andat school levels. The findingsof the studiesprovide initial evidenceof theneed toadapt existing ethical frameworks to the different approaches to learning analytics and to theeducationalcontextstowhichtheymaybeapplied.



Therestof thepaper isstructuredas follows:Section2providesanoverviewoftheethical issuesidentifiedinthelearninganalytics literature;Section3describesthemainaspectsofourapproachto LA; Sections 4 and 5 explain howwe applied our learning analytics approach in university andprimaryschoolcontextsaswellasthemainethicalissuesthatweencountered;Section6discussesandcomparesthefindingsobtainedfromthedifferentcontexts;andfinally,Section7providesthemainconclusionsdrawnfromthiswork. 2 ETHICAL ISSUES IN LEARNING ANALYTICS: A FRAMEWORK OF REFERENCE Asthefieldoflearninganalyticsemergesfromitsinfancy,ethicalissuesrelatedtoitsapplicationtoreal practice are receivingmore attention. In parallelwith the increasing impact of the proposalscomingfromthefield,thereisaneedtoreflectontheconsequencesthattheseproposalshaveonthe persons involved (Slade & Prinsloo, 2013), and on how learning analyticsmust be shaped torespondtothesedemands(Pardo&Siemens,2014).Weaim toprovidenew insightsonhowethical issuesdependon theapproach toLA,andon theparticularcontextstowhichtheseapproachesareapplied.Inordertostructuretheanalysis,thereisa need to organize the issues and establish a common framework that enables discussion aboutthem (Ferguson, 2012). However, to the best of our knowledge, few frameworks yet address theethicalissuesrelatedtotheapplicationofsmall-scaleteacher-ledlearninganalyticsinschools.Thereexiststudiesthatdrawattentiontothepotentialbenefitsof learninganalytics forprimaryschools(see e.g., Ebner & Schön, 2013), or deal with how to support LA-enabled teacher interventions(Wise,2014),buttheydonotusuallyaddresstheethicalorprivacyconcernsthatmayaffectthem.Theexistingframeworkstakeaninstitutionalapproach.Kay,Korn,andOppenheimer(2012)identifya set of legal and ethical issues, with a focus on higher education. Sclater (2014) provides acomprehensivelistofconceptsandissuesrelatedtoethics,accompaniedbyanextensiveliteraturereview.Basedon this review,andafteraprocessof structureddiscussionandvalidation involvingexperts and stakeholders (Sclater, 2015), aCodeofPractice for learninganalyticswas released tosupporteducationalinstitutionsintheeffectiveuseoflearninganalytics(Sclater&Bailey,2015).Intherestofthissection,wedescribethecategoriesdefinedintheCodeofPractice,pointingouthowtheissuescoveredapplytotheteacher-ledsmall-scalelearninganalyticsapproachaddressedinthispaper. Responsibility:Thereisaneedtodefinewhoisresponsibleforthelegal,ethical,andeffectiveuseofLA. Concretely, specific responsibility should be allocated for data collection, anonymization,analysis,retention,andstewardship,aswellasintervention.Attheclassroomlevel,teacherswillbe,inmanycases,theonesresponsiblefortheseaspects.Onequestionderivedfromthisiswhethertheteachersinvolvedhavethecapacityifprovidedwiththetoolsneededtofacethisresponsibility.



Transparency: This aspect refers to how and whether students should be informed about datacollection,results,andthemethodsofanalysis.Anypotentialadverseconsequencesofwithdrawalfrom a learning analytics process should also be explained to them. At the institutional level, itmakes sense to ask whether teachers should be aware of this information, while in small-scaleapproaches, thisquestiondoesnotnormallyapply,astheteacherswillbetheonestocontrol theanalysis.Therefore,themainconcerninthiscaseiswhetherandhowtheenvisioneddatacollectionandanalysisaretobepresentedtothestudents. Consent:Thistopicreferstohowandinwhichcircumstancesstudents(parents)shouldbeaskedforconsent tocollectanduse theirdata,e.g.,aspects related to informedconsent,whetherstudentsholdtherighttooptout,andwhattheconsequencesofdoingthiswillbefortheindividualandforthe group. Consent can be formulated to enable students to decide on the data being collectedaboutthem,ortopreservetheiranonymity.Thistopicalsoincludesissuesrelatedtothe informednatureoftheconsent,askingwhetherconsentcanbeconsideredasinformedwhentheparticipantsdonotunderstandtheanalytics,orwhentheseanalyticsmayevolveinthefutureinunknownways.Theneedtodescribetheanalysissothattheparticipantsunderstanditmaybeveryrelevantattheschool level, where the participants (teachers, children, and their parents)may find it difficult tounderstandwhat isplanned.Attheclassroomlevel,optingoutmaynotbepossible if theanalysisforms part of the learning activities. In caseswhere it is possible, one issue to address is how toprovideteacherswithlogisticalsupporttoexcludetheselearnersfromtheanalysis. Privacy:Accesstostudentdataandanalyticsmustberestrictedtothosewithalegitimateneedtoviewthem.Thereareimportantissuesrelatedtothecapacityoflearninganalyticsalgorithmstore-identify individuals based on the aggregation of these data sources. If there is a need to use“sensitive data” (e.g., ethnicity, religion, sexual preference) additional safeguards and possiblyadditionalconsentshouldbecollectedbeforegrantinganyrequestfromanexternalbodytosharedata. Privacymaybelessimportantattheclassroomlevel,whendataisonlyusedandsharedwithinthe class. However, it is necessary to ensure that privacy is not being violated, especially whenworkingwithminors,wherethisisaverysensitiveissue. Validity:Thisconceptreferstotheextenttowhichtheanalysiscanbetrusted.Relevantquestionsrelatetowhichdatasourcesarenecessarytoensureaccuracy,howtoverifythealgorithmsusedtodraw conclusions, and how to avoid drawing conclusions from spurious correlations. In contextswhere learninganalytics isapplied tohistoricaldata, it isalsonecessary toconsider that studentsdevelop,learn,andgrow,andtheconclusionsbasedonthatdatamustconsiderthatdevelopment.Finally, this aspect refers to amore technical issue related to the authentication of data sourcescoming from public sites or, in general, from third parties, i.e., how to ensure that students arecorrectly identifiedwhenusingdatacoming fromtheseexternaldata sources. Inclassroom-basedexperiences,whereblendedlearningiscommon,therearemanythreatstovalidity,asparticipantsmayinteractinmanyforms,includingfacetofaceoutsidetheclassroom.



Access:Thistopicrelatestowhetherandinwhichwaysstudentscanaccesstheanalyticsthatmakeuseoftheirdata.Thisaspectconsiderstheneedtoallowstudentstocorrectinaccuratedataaboutthemselves.Atthesmallscale,teacherscanaskthelearnersorconsiderextradatasourcesinordertomakecorrections. Enablingpositiveintervention: Thistopicrelatestowhatshouldbedonebytheinstitutionwiththeinformationobtained,andwhattheconsequencesofnotdoinganythingare,e.g.,not informingastudent that s/he risks failing a subject. This aspect also includes issues related to pedagogicalinterventions, suchaswhoshouldbe responsible forpresenting the resultsandhow, thepossibledangeroffavouringonegroupoveranother,howtoadapttheinterventiontothedifferentneedsofthe students, et cetera. The approach discussed in this paper assumes that learning analytics isappliedtosupportteacherinterventionsintheclass.Aswithanyotherpedagogicalintervention,itisnecessarytoreflectontheimpactthattheseinterventions(orlackthereof)haveonthelearners. Minimizing impact: This topic covers some of themain adverse impacts already identified in theapplicationof learninganalyticsandhowtodealwith them.Weshouldbeaware thatananalysiscannever give a completepicture of an individual’s learning andmay sometimes ignorepersonalcircumstances.Thus,wemustensurethattrends,norms,categorization,oranylabellingofstudentsdoes not create bias, reinforce discriminatory attitudes, or increase social exclusion. At the smallscale,thisaspectiscloselyrelatedtotheeffectthattheinterventionwillhaveonthestudents. Stewardshipofdata:Thisaspectisrelatedtotheadministrationofdatabytheinstitution,includingcompliancewith the legal requirements applicable in each case. Data for learning analyticsmustcomplywithexistinginstitutionaldatapolicies,andthesepoliciesmustensurethattherightsofallthe involvedpersonsaremaintained.Additionally,only theminimumdata required todeliver thepurposesoftheanalyticsshouldbecollectedandtheyshouldonlyberetainedforappropriateandclearlydefinedperiods.Teachersrequirehelpinaccomplishingthesetasks.This classification,proposedby theCodeofPractice,wasuseful to structure thediscussionaboutethicalconcernsfoundinthecasesdescribedinSections4and5.Moreover,thisdiscussionhelpedus identifyaspectswhere the framework shouldbeadapted to theapproach to learninganalyticsaddressedinthispaper.Thenextsectionoutlinesthemaincharacteristicsofthisapproach. 3 OUR APPROACH TO LEARNING ANALYTICS Griffiths (2012) identifies two types of analytic interventions with potential impact on teachingpractices: 1) methods oriented to achieve enhanced regulation of the teaching and learningenvironment; and 2) methods and tools intended to help lecturers carry out their tasks moreeffectively.Ourapproachtolearninganalyticsisalignedwiththesecondapproach.Itisdevotedtosupporting teacher reflection, as opposed to prediction, to use the terms of Greller & Drachsler(2012).Thissectionoutlinesthemaincomponentsofourapproach,highlightingthoseaspectsofthe



proposedsolutionswithanimpactontheethicalissuesdiscussedinthispaper.Our learning analytics approach is designed to provide monitoring information to be used forregulation, formative assessment, or self-reflection about the learning design and the learningprocess. We aim to provide teachers with feedback about the accomplishment of pedagogicaldecisionsmadeatdesigntime.Theeducationalsettingswheretheapproachcanbeappliedcoverface-to-faceanddistanceactivitiescarriedoutatdifferentsociallevels(individual,group,andwholeclass). In otherwords, blended Computer-Supported Collaborative Learning (CSCL). In addition, intermsoftechnologicalsupport,theproposalisdevotedtodistributedlearningenvironments(DLEs)made up of an institutional virtual learning environment (VLE), Web 2.0 tools, and GLUE!— anarchitectureusedtointegratethird-partytoolsintoVLEs.Asdescribedintherestofthissection,ourwork was designed to address specific challenges met in these contexts that also have ethicalimplications.Thesechallengesincludedtheneedtosupportteachersintakingresponsibilityfordataanalysis and interpretation; how to help them with stewardship of data in these complextechnological systems; as well as how to enhance data validity in blended learning supported byDLEs.

Figure1:Overviewofthemaincomponentsinvolvedinourlearninganalyticsapproach.

Thesolutionconsistsofthreecomponents(SeeFigure1):twoprocessesandanarchitecturefordatagatheringandintegrationinDLEstohelpteachersintegratelearninganalyticswithintheirpractice.The first component isamonitoring-awaredesignprocessof the learning scenario that takes intoaccount the teacher’s information needs (Rodríguez-Triana et al., 2013). This enriched designprocess helps teachers identify andmake explicit which moments or aspects of their envisionedlearning activities should bemonitored. These aspects are based on the identification of a set ofconstraints: special conditions that must be met to comply with the pedagogical intentionsexpressed in thedesign, e.g.,whetherparticipationof all the students in a group is required in acertain phase of the activity, or whether a particular product must be delivered at a specificdeadline, et cetera. The process entails a second phase, during which teachers are prompted to



enrich thedesign to satisfy thesemonitoringneeds (e.g., choosing themost appropriate tools forbothpedagogicalandmonitoringconcerns,oridentifyingcomplementarydatasources).Thesecondcomponent of the solution is amonitoring process guided by the decisionsmade at design time(Rodríguez-Trianaet al., 2012). In thismonitoringprocess, thedata gathering is focusedon thosesourceschosenbytheteacherandthedataanalysisisdesignedtoverifywhetherthecurrentstate(the gathered evidence) matches the desired state (the learning design). These two processes(designandmonitoring)aretwosidesofanoverallapproachthataimstohelpteacherstakecontrolof the data analysis and interpretation. By means of the design process, they become activeparticipantsinthedefinitionoftheanalysis.Thecontextualizedvisualizationoftheresultsenabledbythescript-awaremonitoringprocessisdesignedtosupportteachersintheirinterpretationoftheanalyses. ThethirdcomponentofourproposalisGLUE!-CAS,anarchitecturethataddressestheneedofdatagathering and integration in DLEsmade up of VLEs (typicallyMoodle orMediawiki) andWeb 2.0tools (e.g., Google applications) (Rodríguez-Triana et al., 2011). GLUE!-CAS defines how to collectand integrate data coming from these data sources, overcoming the problems of gathering datafromexternaltoolswhenusingaVLE.However,inblendedscenarios,theseautomaticdatasourcesarenotenoughtogetafullpictureoftheinteractionstakingplace.Frequently,partofthelearningprocess occurs outside the technological context. Besides, the information and communicationtechnologies (ICTs) register a limited set of evidence, usually based on user interactionswith theplatforms,andsometimes it isnotpossible toauthenticate thestudent identityproperly (Slade&Prinsloo,2013).SincerestrictingtheanalysistothedataregisteredbytheICTtoolsmayofferonlyapartialviewoftheuseractivity(Avouris,Fiotakis,Kahrimanis,Margaritis,&Komis,2007),GLUE!-CASenablesthe integrationofad-hoc informationprovidedbytheparticipants inthe learningcontext.This practice enriches the evidence gathered and allows teachers to triangulate the data comingfromthedifferentdatasources.Fromanethicalpointofview,thisapproachenablesteachersandstudentstocorrectthedataautomaticallycollectedfromtheICTtools(Sclater,2014)andtobetterunderstand the reasons behind the results obtained (Greller & Drachsler, 2012). Overall, thisintegration of heterogeneous data sources is designed to increase the validity of the analysis. Intermsofprivacy, this involvementof thestakeholdersalsohasbenefits, letting themdecidewhatinformation they want to share (Slade & Prinsloo, 2013). Finally, the technical solution helpsteacherstofacethedifficultiesassociatedwithdatastewardshipinDLEs.To support teachers in their analysis,we implementedGLIMPSE (Rodríguez-Triana et al., 2013), atoolthat, interactingwithGLUE!-CAS,automatesdatagathering, integration,andanalysis,offeringtheteacheracomparisonbetweenthecurrentandthedesiredstateofthe learningscenario.Theproductof this tool is amonitoring reportwhere the information is visualizedbymeansof tablesthatconnecttheparticipants,thedatasources,theindicators,andthewarningsthatemergedfromtheanalysis(seeFigure2belowforanexample).



4 FIRST SCENARIO: HIGHER EDUCATION Theproposalpresentedintheprevioussectionwasiterativelyappliedtosevenlearningscenariosinhigher education (Rodríguez-Triana et al., 2015). In this paper,we focus on the last two of thesescenarios,whichweredesigned toevaluate theproposal in its finalversion.This sectionoffersanoverview of the learning contexts aswell as how the proposal was implemented throughout thelearningscenariolifecycle.Wewillnotdescribetheresultsrelatedtotheapplicationofthelearninganalyticsapproachassuch,whichhavebeenreportedelsewhere(seeRodríguez-Trianaetal.,2015).Wewill focuson illustratingthoseaspectsof thecases thathadan influenceontheethical issuesdescribedinSection2. 4.1 Learning Context TheapproachdescribedinSection3wasappliedtotwolearningscenariosinhighereducationwitha common profile: 3–4 weeks, implementing learning designs inspired by CSCL principles andsupportedbyDLEs,interweavingface-to-faceanddistanceactivitiesaswellasblendedinteractionsamongstudents.Thestudiesinvolvedtwoteachersfromdifferentbackgrounds,withdifferentlevelsof expertise in CSCL scenarios, and different knowledge about the proposal. To reference thesestudies,wewilluseHE1andHE2aslabelsforthefirstandsecondscenariosinhighereducation. Thefirststudy(HE1)wascarriedoutinanEarlyChildhoodEducationcourse,involvingateacherwhowasnotexpertinCSCLscenariosandwhohadlessthan6yearsofteachingexperience.Ofthe165studentsenrolledinthecourse,150participated.Thelearningscenariolasted4weeksandconsistedof various distance and face-to-face activities combining individual and collaborative work. Thewhole learning designwas supportedbyMoodle andGoogle applications, and involved a total of316resources.Themainchallengeof this scenariowas tocopewith thehighnumberof studentsandresources. Thesecondstudy(HE2)tookplaceinaneducationalresearchcoursethatformedpartofaMaster’sdegree forpre-service secondaryeducation teachers. The teacher in chargeof this coursewasanexpert in CSCL and had previous knowledge of the approach. Over 3 weeks, 15 students wereinvolved in this study. They worked on defining a proposal for an educational research project,combining individual, group,andclassactivities,aswellas face-to-faceanddistance learning.Thewhole learning process was technologically supported by means of MediaWiki and Googleapplications,requiring77filesintotal.Themainchallengeofthisscenariowasthecomplexityofthedesign: many interrelated activities occurring in a short period of time that demanded muchattentionfromtheteachertoavoidproblemsthatcouldjeopardizethescenario. 4.2 Application of the Proposal The application of the proposal started with the design of the learning scenarios, following the



monitoring-awaredesignprocess (see Section3).This process is designed to give responsibility ofthelearninganalyticsprocesstotheteacherandprovidesameansofaddressingthevalidityoftheanalysis. Besides paying attention to the pedagogical aspects of the learning activities, thedesignprocesssupportedtheteachers’articulationoftheirmonitoringconcerns.Basedontheinformationprovided in the design, and following the proposed design process, the teachers were informedaboutthedataavailabletoverifytheconstraintsoftheenvisionedlearningscenario,whethertherewasenoughevidence toevaluate them,andwhichcomplementarydatasourcescouldbeused toimprove the analysis. This information gave the teachers the opportunity to reflect on themonitoringprocessandenhanceit. Inbothscenarios,theteachersdecidedtotriangulatethedatacomingfromtheICTtoolswithdataprovidedad-hocbythestudentsandtheteachersthemselves.Theseaspectsareillustratedinthereflectionsmadebybothteacherswhentheywereinterviewedafterthedesignphase:

Ithashelpedmetofirstreflectaboutwhichwerethemostconflictivepointsofthedesign.Oncetheywereidentified,andknowingwhichtechnologicaltoolsIwasgoingtouseineachphase, I couldknowbothwhatkindof information theywouldprovideandwhat thebestmoment to get that information would be. [...] In the original design, I did not foreseecontrollingclassroomattendanceinasystematicmanner,noradministeringaquestionnairetofindoutthecollaboration,taskdistribution,andworkloadaspectsforeachofthephases.[...] Thedesign processmademe appreciate the importanceof including new informationsources to enable gathering additional evidence. [HE1, Teacher interview after the designphase]

In case the tools that I had inminddidnot providemonitoring information, Iwouldhavesubstituted themwithother tools (provided theyhave similar functionality to support thestudents’work,ofcourse).[HE2,Teacherinterviewafterthedesignphase]

Thus,throughoutthemonitoring-awaredesignprocess,theteacherswereresponsibleforthedatagathering and analysis, definingwhich constraints needed to be evaluated during the enactment,selectingthedatasources,theinformationtoberetrieved,andwhenitshouldbecollected. A second ethical aspect addressed in the proposal regards transparency and consent. Once theteachersdefined theanalysis tobecarriedout, the studentswere informedabout thedata tobegathered,themetrics,thepurposeoftheanalysis,anditspotentialimpact.Wethenaskedthemtogive their permission fordata collection to takeplace: almost all the studentsparticipating in thelearningscenariosagreedtothis(150outof165inHE1andall15inHE2).Thetoolssupportingthedesign process allowed us to specify which students were taken into consideration, focusing theanalysis on those students who had given their consent. Therefore, our proposal was able toaddress,at least froma technicalpointof view, theproblemsassociatedwith letting learnersoptoutfromtheanalysis. Athird issuerelatestothestewardshipofdata.Duringthescenarios,thestudents’dataregardingthe actions registered on the different platforms (e.g., accesses, editions, uploads, et cetera)was



gathered from the technological environment. The data came from the VLEs (Moodle andMediaWiki), the Web 2.0 tools (e.g., Google Apps), and the architecture that supported theintegrationofthetoolsintheVLEs(GLUE!).Ontheonehand,boththeVLEsandGLUE!werehostedonourownservers.Thus,wedidnothave todealwithpermission issues inorder toconnect thedatagatheringtool(GLUE!-CAS)withtheseplatforms,whichnormallyconstitutesanobstaclewhenworking with institutional or third-party platforms. On the other hand, GLUE!-CAS allowed us tocollect the data from the third-party Web 2.0 tools integrated in the DLE. Thus, even if theinformation was initially controlled by the owners of these tools, we made it available for theteacherstousefortheiranalyses.TheintegrationofthesedatabyGLUE!-CASenabledteacherstocontrolthesedata,anduseitevenwhenitwasremovedornolongeravailableintheoriginaltools.The integration of data inGLUE!-CAS is also important to increase thevalidityof the analysis. Asmentionedabove,thisfocusonvalidityisaddressedbythedesignprocess,whichpromotesteacherreflectionontheaccuracyofthedataandenablesthemtoincludenewdatasourcestoenhancethisvalidity. Following thesedesigndecisions, teachers and students contributed activelybyprovidingtheir own data. The teachers provided information about student attendance at the face-to-facesessions togetherwitha fewnotes,and thestudents,bymeansofquestionnaires,describedhowtheyhadparticipatedinthelearningactivities(howtheyhadinteracted,whichtoolswereused,howmuchtimetheyhaddevotedtothetasks,etcetera). Via GLUE!-CAS and GLIMPSE, data generated by the multiple data sources were collected,integrated, and analyzed according to the teacher’s decisions. Based on the data gathered, theteachers received the corresponding monitoring reports taking into account the relevant datasources,theactionstobelogged,thetimeframes,andtheindicatorstobeverified.Error!Referencesource not found. shows one of thesemonitoring reports sent to the teacher (only three groupsappear due to constraints of space). The columns show, from left to right: the groups and theirmembers, the data sources employed in this case, and the warnings issued by GLIMPSE when acondition specified at design time is not met. The cells coloured in green refer to students oractivitiesthathavecompliedwiththespecifiedconstraints,whiletheredcellshighlightthepointswherethereisnoevidencesupportingtheteacher’sexpectationsaboutthestudents’ involvementortheuseofresources.Forexample,Figure2showsthatnoaccessbyStudentName6totheFinalresearchproposalhadbeenregistered.Theteacherhadspecifiedatdesigntimethatthisresourcehad to be used at least once by each groupmember, as the lack of access could imply a lack ofparticipation of thatmember in the authorship of the report. This problemwas visualized in thereportbymeansoftheredcellcorrespondingtoStudentName6,andthewarningthatappearsontheright-handsideofthetable.Ascanbeobserved,thereportssummarizedtheevidencecollected,highlightingpotentialproblemsthatneededtobechecked.



Figure2:ExcerptfromthemonitoringreportsenttotheteachertheHE2study(anonymized

version).

The integrationofmultipledata sources in theanalysisplayeda crucial role in the successof theproposal. First, it significantly increased theaccuracy of the results. Complementing the evidencefrom the ICT toolswith the teacher’sobservations and the students’ feedbackallowedus to takeintoaccountnotonlythecomputer-mediatedinteractionsbutalsotheface-to-faceones.

The students’ reporthasbeenveryuseful, inorder to take intoaccount theirperspective.[...] Being able to incorporate my notes with the comments received from students hassimplifiedmyworkagreatdeal.[...]Havingallthedatagatheredinoneplacesimplifiesthemonitoring.Havingtheinformationcentralizedhelpsavoidmisunderstandingsandkeepsthesituationundercontrol.[HE1,Teacherinterviewaftertheenactment]The integration isnotonlyuseful, it isnecessary. Itgivesyouacomplementaryviewofanactivity that happens in different settings/moments. It uses blended sources to informblendedactivities.[HE2,Teacherinterviewaftertheenactment]

We evaluated the validity of themonitoring reports by comparing the results obtained with thecomplementary teacher observations, the additional student comments, the researcherobservations,andthelearningoutcomesinthetools.Despitetheexistenceoferrors,themonitoring



reportsprovidedaperspectiveofthelearningprocessthatwasclosetotherealfacts(in99.67%and97.81%oftheevaluatedconditions).Aspreviouslydiscussed,ourapproachhandsresponsibilityfortheanalysis(boththedesignandtheinterpretation)toteachers.Oneaspectthatmustbeaddressedregardingresponsibility iswhetherteachersareable toappropriate theanalysesandactupon them.Ascanbeseen inFigure2, theinformation provided to teachers was direct and contextualized in their designs. Both teachersagreedthatinterpretingthereportscouldbedonequicklyanddidnotpresentanyproblems.

Interpretingthemonitoringreportswasveryeasy.Withonelook,Iknewwhethertherehadbeenanyproblems.[...]Idedicated10minutesatmost:5minutestoreadeverything,plusanother 5minutes to take the correspondingmeasures. [HE1, Teacher interview after theenactment]Interpreting the reportswas simple and immediate. The informationprovided is clear anddoesnotleadtomisinterpretations.[HE2,Teacherinterviewaftertheenactment]

During the learning scenario, the teachers used the information to identify potential problems aswell as to intervene and regulate the scenario when appropriate. Afterwards, once the activitiesended,theteachersemployedthemonitoringreportstosupportreflectiononthelearningprocess.

WhenIdetectedaproblem,Icontactedthestudent.Infact,Ihavesentquiteafewemailswithwakeupcalls,reminders,andaskingwhathappened.Thisissomethingwenormallydonotdobecausewedonothavemeansto followthestudents’workclosely. [HE1,Teacherinterviewaftertheenactment]The monitoring reports required regulatory tasks only in two cases. In general, allstudents/groupsfollowedtheplan,exceptforthedevelopmentofthereportinsmallgroups(whichIcommentedonverballywiththemintheclassroom)andinthelastactivity,inwhichthreestudentsforgottosendthepeer-assessmentreport,andthusIhadtosendareminderto themviaemail. Itwascriticalbecauseneither theynor Iwouldhave remembereduntilthefinalassessment,whichcoincidedwiththeendofthecourse.Therestservedtocheckthateverythingwasgoingwell.[HE2,Teacherinterviewaftertheenactment]

Regardingstudentaccesstothedataanalysis,weshouldrememberthatourapproachisdevotedtosupporting teachers. Thus,during the scenarios studentsdidnotautomaticallyhaveaccess to themonitoringreports.However,themessagessenttotheteacherinthereportsmadeitclearthattheresultshadtobechecked.Therefore,anynon-expectedresultshouldhavebeendiscussedwiththestudent, who could clarify the reasons behind it. The teachers’ comments during the interviewspresentedearlierillustratehowtheyproceededinsuchcases.Insummary,thetwouniversitycasesillustratehowourclassroom-basedlearninganalyticsapproach



provides teachers with tools to become responsible for data analysis; how data stewardship wassolvedbytheuseofGLUE!-CAS(andthefactthatwecoulduseourownserversattheuniversity);howwedealtwith theneed toprovide transparent information to theparticipants; andhow thetechnicalsolutionenabledteacherstoexcludefromtheanalysisthosewhodidnotsigntheconsentform.Wehavealsoshownouremphasisonthevalidityoftheanalysisbymeansoftheintegrationofseveraldatasources,plannedatdesigntimeandaddressedtechnicallybyGLUE!-CAS;andhowtheresultswereshownbyGLIMPSEinacontextualizedmannerthatteacherscouldappropriatetointerveneinthelearningsituationswhenneeded.Thestudentsdidnotreceivethedatadirectly,butteacherswerepromptedtocompareresultswithotheravailable informationbeforeacting,asthewholeproposalmakesteachersawareof theweaknessesassociatedwithdataanalysis inblendedscenarios. 5 SECOND SCENARIO: PRIMARY SCHOOL AswedescribedinSection3,ourproposalwasdesignedtohelpteachersinblendedCSCLscenariossupportedbyDLEs.Althoughallourpreviousstudieshadbeencarriedoutinhighereducation,werealizedthatatothereducationallevels,teachersfacedsimilarproblems.Thus,wesetouttoapplyour proposal in a primary education settingwith the aim of checking its adaptability to this newcontext,andtounveilthepotentiallimitationsandconstraintsimposedbysuchalearningcontext.Thissectionprovidesabriefdescriptionofthelearningcontextincludingananalysisofthedifferentethicalissueswefacedduringthestudy. 5.1 Learning Context The case studywas run in a first grade class (6–7 yearsold)with24 students at a rural school inValladolid,Spain.Thisschoolhas300studentsand29teachers.Theteacher involved inthisstudyusedBlogger inhisclasses incombinationwithexternalWeb2.0tools(suchasYouTubevideosorEducaplay games). In spite of a general positive experience with this approach, one of his mainconcernswasthattheseblogsdidnottellhimwhohadreadoraccessedthepages.Thisprecludedhimfromknowingwhetherthestudentswereusingthelearningmaterialsofferedintheblog. Inordertoaddressthisproblem,weusedGLUE!tobuildtheDLE.BloggerwastreatedastheVLEwhere the third-party tools (e.g., the videos and games)were embedded.With this technologicalsetting, it was possible to trace student activity during the learning process and then inform theteacherabouttheiruseofthetool. The intervention ran between 14May and 2 June 2014. The teacher proposed two lesson plansinvolvingablogandseveralexternalresources,inwhichtheteacheraskedthestudentstocarryoutsome activities at home such aswatching a video, reading an online text, or playing educationalgames.



5.2 Application of the Proposal Aswiththepreviousstudy,duringthisdesignprocess,theteacherdescribedthelearningactivities,chosetheresources tobeused,andconfiguredtheways inwhich the learningscenariowouldbemonitored: he decided which constraints should be checked, specified student actions to bemonitored,aswellasthetimeframesanddeliverydatesofthemonitoringreports. Oncetheteacherconfiguredtheanalyses,weinformedthedifferentparticipants,namelytheschoolprincipal and the parents/legal guardians of the students, since they were not adults. A detailedexplanationofthedata,metrics,purpose,andpotentialimpactoftheanalyseswasgiventothem.Boththeschoolprincipalandthefamiliessupportedthiskindofinnovativepracticeandgavetheirconsentforthedatacollection.Regarding thecollectionof identifieddata,weencounteredanunexpected legalobstacle.Bloggercanbeconfiguredsothatonlyregisteredusersaccesstheblogs,butthisrequiresaGmailaccount.However,6–7yearoldchildrenarenot legallyallowedtoownanemailaccountonthatserver. Inorder toovercomethisobstacle,wehadto look foraworkaround,askingthe families toopenanemailaccountonbehalfoftheirchildren,andthenaccessthesiteusingthataccount.Evenso,wedidnot find it difficult toobtain consentand family support to createanduse thedummye-mailaccounts. However, we must note that obtaining consent when working with minors is a verydelicateissueandmustbecarefullyaddressed. Another issueweencounteredrelatedtothekindofdatathatcouldbecollectedandanalyzed. Inuppereducationallevels,theinputandinteractionswithtechnologiesmaybeveryrich.However,attheageof6to7,studentsfrequentlydonothavetheskillstocarryoutverycomplextasks(e.g.,thestudentswrote littlebefore theendof thecourse). Thus, thekindofdataavailable is likely tobelimited,andmanyonlineeducationaltoolsdonotofferanydataaboutthe learner’swork. Infact,the data gathered from the technological environmentmainly consisted of the actions logged byGLUE!, specifically student accesses to theWeb 2.0 tools. This information was easily accessiblebecause GLUE! was hosted on our servers and we could access the systems without additionalbureaucracy. Whilethelearningscenariowastakingplace,theteacherreceivedmonitoringreportsaccordingtotheplandrawnup at design time.Figure 3 shows a piece of information from thosemonitoringreports.Thesereportssimplypresentedtheevidencecollected (accesses to thetools)highlightingpotentialproblemsthatneededtobechecked,e.g.,studentswhohadnotaccessedthetools,andtherefore,couldnothaveusedtheresources.Themonitoringreportpresentedinthefigurereferstothenumberoftimesthestudentsaccessedtheactivitiesinoneofthemonitoredlessonplans.Itcanbeseenthatonlythreestudentsof24completedalltheactivities,whilenineoutof24didnotaccessany.Thesamekindofvisualizationwasshowntotheteacherforthesecondlessonplan.Theresults relating to the participation of the students were similar. Only five students out of 24



completed all the activities,while 11 out of 24 did not access any. When the researchers sharedtheseresultswiththeteacher,hefeltconfused,aswecanseeinthefollowingextract:

Honestly, I did not think this could happen. I mean, I thought that the majority of mystudentscouldcompletetheactivities.Ithinkitisimportanttocommunicatetheseresultstothefamilies.[PE,Teacherinterviewaftertheenactment]

Figure3:Outputshowntotheteacheraboutthenumberofstudentswhoaccessedtheactivities

inLessonPlan1.

During the face-to-face sessions, the teacher asked the students whether they had done theirhomework in order to compare their answers with themonitoring reports. The information wasgenerallyaccurateandhelped the teacher to interveneappropriately.However,oneof the issuesthat we faced related to the students’ virtual identities. As noted by the teacher in the secondinterview,notall families knewhow to solve someof the technicaldifficulties, and some familiesaccessedthesiteusingtheparents’ownaccounts.

When I met with the families, many of them toldme that they had hadmany problemsaccessingtheactivitiesembeddedintheblog.TheysaidthattheyhadreadandfollowedalltheinstructionsIgave,butstillfounditimpossible.Moreover,InoticedwhenIreviewedthemonitoringreport,thattherewerefamiliesthathadaccessedtheactivitieswithadifferente-mailaccount.[PE,Teacherinterviewaftertheenactment]

Obviously,thetechnicaldifficultiesencounteredbythefamilies,togetherwiththeworkaroundwehad to use to be able to identify the children using surrogate accounts, resulted in a case of“enmeshedidentities”(Greller&Drachsler,2012).Awareofthisproblem,inthefinalinterview,theteachersaidthatifheweretouseasimilaractivityinfuture,hewoulddevotemoretimetotrainingthefamiliestoaccesstheon-lineactivitieswiththee-maillogin,aswellassupervisingtheirusage.



Inspiteofthesimplicityoftheprojectandalltherestrictionsrelatedtothequalityofthedata,theteacherwasverypositiveaboutthefeedbackprovided:

Itisimportanttoprovidestudentswithotherwaysoflearning,aswellastoassesslearning.Icansaythattheseteachingandlearningpracticesare innovativeforstudentsandwehavetheopportunity tomotivate themandengagethe families in theeducationalprocess. [PE,Teacher’sreflectivediary]

In addition, the teacher identified several potential uses of the analysis. Among other uses, hehighlightedknowinginadvancewhetherthestudentshaddonetheirhomework,beingabletosendreminders,identifyingmore/lessattractiveresources,detectingwhichdaysoftheweekthestudentsweremoreactive,analyzethestudents’developmentinrelationtotheirhomework,etcetera. Regarding student access and privacy,we should recall that the teacher is the target user of theapproach presented in this paper. Nevertheless, the results, especially in the case of unexpectedbehaviours,shouldbecheckedwithstudents/parentswhocanclarifywhathappened.Especiallyinthis case, themeaningof the reportsmightnotbeeasilyunderstandableby childrendue to theirage.Buttheteachercouldshareitwiththeparentsorwiththeinstitutionitselfas“evidence”ofthelearningprocesstobediscussed.Therefore,teachersareinchargeofsharingthedataanalyseswiththosewholegitimatelyneedtoviewthem,andthissharingmusttakeintoaccounttheprivacyandlegalissuesapplicableineachcase. 6 DISCUSSION The studies presented in Sections 4 and 5 illuminate our discussion about how the currentframeworks related to ethical issues in learning analytics apply to the analytical interventionsweenvision, as well as how they depend on the educational contexts where we applied them.Weidentifythreelevelsofdiscussion:1)implicationsrelatedtothelearninganalyticsapproachtowhichthisworkbelongs, i.e., small-scale analytics tohelp teachersmanage their classes; 2) implicationsderived from the actual solutions we propose to support this learning analytics approach, and 3)issues related to theeducational contextswhere the solutionsareapplied.After thediscussionatthese three levels, we drafted a set of guidelines that could be taken into account by teachersaimingtoconductlearninganalyticsscenariosofthekinddiscussedinthispaper. 6.1 Implications Related to the Teacher-Centred Classroom-Based Approach to LA Asmentionedabove,ourapproachtolearninganalyticsisorientedtosupportingteacherreflectionin order to help them with the management of their classrooms. This is one of the possibleapproaches to learning analytics identified by Griffiths (2012). The other two approaches are 1)institutional approaches that seek to increase the efficiency of the institution; and 2) approachesassociatedwiththeenhancedregulationoftheteachingandlearningenvironment.



As mentioned in Section 2, the framework used to structure our discussion responds to aninstitution-ledapproach.This focuscanbeobserved in thekindofquestionsposedtoanalyzetheconcepts,and in the responsiblestakeholders identified.Thequestions included in the frameworkfocus on what the role of the institution and/or the students in the overall process of learninganalytics is.Teachersappearonlyassecondaryactorsor—marginally—aspotentialprovidersofdata. Incontrast, inourapproach the teacherplays the roleof the institution— incontrolof thedataandtheanalysis—andofthereceiveroftheinformation—role-playedbythestudentsinthereferenceframework.Dueto thesedifferences,wehadtoadapt thequestionsposed in the framework tobetter fitourapproach. This translation was straightforward. However, the need for adaptation could be anobstacle for a practitioner whowanted to employ the framework to analyze a learning analyticsintervention of the kind discussed in this paper. This justifies the need to refine the ethicalframeworktoadaptittothecharacteristicsofthelearninganalyticsapproach.Afirstelementofrefinementwouldbetheinclusionofteachersasresponsiblestakeholders,andofquestions related to their role as data providers and receivers. A second element regards theemphasis given to different aspects of learning analytics. For example, while privacy issues areparamountinaninstitutionalapproach,theyhavedifferentimplicationsinaclosedsystem,wheretheteacherusesalreadyavailabledataandsharesitwiththestudents.Infact,inthesecases,newethical issuesmayarise,more related to classroomorchestration,which shouldalsobe identifiedandincludedintherevisedframework. 6.2 Implications of the Design-Aware Learning Analytics Approach Our approach to learning analytics is strongly based on the participation of the teacher from thebeginningof the lifecycleof the learningactivity. It takes intoaccount theneed to integratedatafrom different sources — including data provided by teachers and students — that offer acomplementary view to the evidence recorded by the technological infrastructure. This design-aware learning analytics approach helps to address a number of ethical issues common in thesecontexts.First, the approach has a strong impact on the validity of the data in these blended learningscenariosbasedontheuseofDLEs.TheintegrationofdifferentkindsofdatabymeansoftheGLUE!-CASarchitecture,enriched theanalysis. The fact that thedesignprocesshelps teachers reflectonthemissingdataandenablesthemtointroducedataprovidedbytheparticipantsalsoincreasedthevalidity. The results of the two cases show that the output given to the teachers, based on thiscombinationofdata,wasaccurateinmostcases.Thesecond issueaddressedbyourproposal refers toresponsibility for theanalysis.Theapproach



helps teachers takedecisions about thedataand theanalytics thatwill beapplied to the setting.Thisschemamakesteachersfullyawareofthepurposeoftheanalysis,andavoidspossiblenegativeeffectsofsystemsthatdemandnewcompetenciesofteachersinordertointerprettheresults.Thepossibilityofadverseimpactisminimizedbecause,inourcase,theanalysisdoesnotentailanynegativeeffectonthestudents.Indeed,thewaytheapproachispresentedtoteachersmakesthemrealizethattheinformationavailableisnotcomplete,andtheyareresponsibleforaskingorlookingfor other data sources to complement it. Thus, no decision is made automatically based on theresults. The analysis only aims to detect potential problems in order to address them as soon aspossible. To bemore precise, it is the teacher who, aware of the information, should verify thesituationandinterveneifnecessary.6.3 Issues that Depend on the Educational Context Ourexperienceinapplyinglearninganalyticstotwodifferenteducationallevels(seeSections4and5)showedusthatethicalissuesalsodependonthekindoflearninganalyticsprocessesthatcanbeexpectedinthesecontexts.Ashighlightedbythecasesdiscussedinthiswork,wefoundnewlegaland ethical concerns in the primary school context that we had not met in higher educationscenarios.Afirstissueaffectedbythischangeofcontextreferstothestewardshipofdata.Inhighereducationsettings,itisusualtheinstitutionthatownstheserverswherethelearningenvironmentsresideandthe activities take place. Teachers can access these platforms, and count on technical staff toconfiguretheanalysis.Onthecontrary,schoolsoftendonothaveeasyaccesstothiskindoflearningplatform, or, if they do, they do not have access to technical staff who can support the analysis(Blackwell,Lauricella,Wartella,Robb,&Schomburg,2013;Lim,Zhao,Tondeur,Chai,&Tsai,2013).Due to theseproblems, teachersand schoolsare increasingly relyingoncloudandWeb2.0 tools.Thedependencyon theseexternalWeb2.0 toolsposesnew legalproblemsandethical issues, aswasillustratedinourschoolcase.Students(minors)cannotlegallyhavetheirownaccounts,leadingto a conflictwith the need for learning analytics processes to identify users. One solution, aswedescribed,istorelyonthefamiliesofthechildrentoobtainasurrogateidentity,butthisaddsmorecomplexitytotheanalysisandnewthreatstodatavalidity.Inthecasediscussedinthispaper,somefamilieswerenot able tohelp their children toperform the tasks,whileothersused theparents’identitiestoaccessactivities.Inbothcases,thishadaneffectonthevalidityofthedatacollectedanddisplayedtotheteacher.Asdiscussedintheprevioussection,theteacherknewinadvancethelimitationsoftheimplementedapproach, and did not take any direct action based exclusively on the monitoring reports.Nevertheless, he found the information provided useful, as it gave him a general overview abouthowthestudentsworkedathomewiththevirtualenvironmentthathehadsetup.



Whenworkingwithlowerlevelcourses(likethefirstgradersstudiedinthispaper),childrenarenotautonomousintheiruseofcomputers,andfamiliesmustbeconsideredactorstotakeintoaccountatmanylevels(permission,training,collectionofdata,etcetera).Theinclusionoffamiliesrequiresareflectiononhowthischangeaffectstheotherethicalissuesdiscussedinthefield.Forexample,theoptionofshowingtheresultstothestudents—andthereforetotheirfamilies—mustbebalancedagainstissuessuchasprivacy,transparency,andaction,astheseareboundtobeanissuewhenthefamiliesareawarethatthetaskscarriedoutathomearebeinganalyzedandusedbytheteachers. Afinal issuerelevant fortheapplicationof learninganalytics inschoolcontexts is thesimplicityoftheinteractionsbetweenthechildren—especiallyfirstgraders—andthesystem.Thisposessomequestions about the kind of inferences that can be drawn from this very basic information. Inprinciple, considering the limitations imposed by the data available, a simple learning analyticsapproach showing basic data to the teacher seems more coherent than sophisticated analyses.However,futuredevelopmentsinthefieldcouldchallengethisview.Overall, our analysis suggests that the work being done by the learning analytics community todefine a general framework or code of practice for learning analyticsmust take into account theapproach and the educational contexts where learning analytics is applied. Further work onanalyzing the particular characteristics common to the different approaches and educationalcontextswouldhelptoidentifythesenewaspectsandtorefinetheproposedframeworks.6.4 Proposals for an Expanded Ethical Framework for Learning Analytics Asaconclusionofourreflection,weproposeasetofrecommendationstobetakenintoaccounttoextendorcomplementtheexisting frameworksofethical issues in learninganalytics.Asdiscussedabove, the existing frameworks are oriented to institutions and therefore apply to large andmedium-scaleinstitution-ledlearninganalyticapproaches.Ifweconsiderthatlearninganalyticscanbe applied in small-scale scenarios, there is a need to adapt the existing frameworks to theseapproachescurrentlycoexistinginthelearninganalyticsfield.Apossiblewaytoaddressthisgoalistodefinedifferentitinerariesdependingontheapproachtolearninganalytics,sothatpractitionerscanfocustheirviewonthequestionsmostimportanttotheirapproach.Asimilarapproachistakenin Kay et al., (2012), defining the different legal and ethical priorities in six use cases related tolearninganalyticsinhighereducation.Oneoftheseitinerariescouldbedefinedforthelearninganalyticsapproachdiscussedinthispaper,i.e.,classroom-basedlearninganalyticsorientedtosupportingteachers’regulationoftheirclasses.In order to provide an initial proposal, we have adapted the questions posed by Sclater’s (2015)framework to produce a set of guidelines that could be taken into account by teacherswilling toapply learning analytics from this perspective. These guidelines arepresented in Table 1, orderedusingthecategoriesemployedinthecurrentCodeofPractice(Sclater&Bailey,2015).



ContrarytotheframeworkproposedbySclater(2015),theserecommendationsassumethat,inthiscontext, teacherswill take responsibility formostof theaspectsmentioned inTable1.Therefore,teachersmust appear as amain stakeholder in the itineraries addressing these learning analyticsapproaches.

Table1:Listofrecommendationsforsmall-scaleteacher-ledlearninganalytics.

Category RecommendationConsent • If information is already being tracked, inform the students (or families) about it,

otherwise, ask for formal consent/agreement before data can be collected and/oranalyzed.

• Beexplicitaboutwhatyoumightdowith that informationand, ifpossible,agreeonthesepossibilitieswiththestudents.

• Provide students with the option to update their digital dossiers and provide extra(possiblyqualitative)databuttriangulateittoverifytheydonot“game”thesystem.

• Datashouldbedeletedwhenindividualsnolongerwantittobeprocessedorwhenitisnolongerofuseforitsoriginalpurpose,asisthecasewithanyotherstudentdata.

• Reflect on the consequences that opting out of the analysis will have on theparticipants(e.g.,lackoffeedbackduetothelackofanalytics)andinformthem.

Transparency • Beexplicit aboutwhichdata is collected, how it is interpreted,why, andhow itwillaffectthelearningprocess.

Access • Reflectonwhetherthestudentsshouldaccessthedataheldaboutthem,theanalysisof thedata, the labelsattached to them,and if so, inwhich format this informationshouldbeprovidedtothem.

• Considerthepossibilityoflettingstudentscorrectthedatastoredaboutthem.

Responsibility • Youareinchargeofinterpretingandvalidatingtheanalysesaswellasdecidingwhattodobasedontheanalytics(e.g.,howtoregulate,intervene,etcetera).

Privacy • If some data must be anonymous, be sure it cannot be re-identified by contextualinformationavailabletotheusers.

• Ifyouusedatafromexternalsources,(e.g.,Web2.0tools)besureyoucanidentifytheowner correctly, and that other ethical and privacy issues are not put at risk whenusingthosesources.

Validity • The evidence obtainedmay be incomplete. Try to involve students (and families) inordertoincreaseitsaccuracy.

Stewardship • Usethedataneededfortheanalysis,nothingmore. • Be sure that you complywith the data protection laws applicable in your region or

country. • The data should be preserved, secured, and shared in the same way as any other

studentdata. Avoidingnegativeimpact

• Consider that your analysis may produce unexpected results that prompt you tointervenewiththestudent.Whichkindsofobligationswillinfluenceyouractions?



The recommendations presented in Table 1 should be considered as a first attempt to structurereflectiononethicalissuesandlogisticalconcernsinsmall-scaleteacher-ledlearninganalytics.Theyshouldbe subject to refinementby their application toother cases, andbypublicdiscussionwithexpertsinthefield.Thesediscussionsmighttakeplaceinthecontextofrecentinitiativespromotedbyvariousgovernments(Hylen,2015)thathighlightthecreationofpoliciesandsharingpracticesoflearninganalyticsinschools. 7 CONCLUSION Interestinaddressingethicalissuesinlearninganalyticsisstartingtoflourishintheformofethicalframeworksthatguidecodesofconductforpractitioners.Theseframeworksareusefulinstrumentsto structure the discussion and promote a more mature application of LA. However, currentframeworks are focused on institutionally oriented higher education learning analytics. There is aneed to extend and adapt these frameworks to the characteristics of different learning analyticsapproaches and educational contexts. Through two studies in primary and higher educationcontexts, this work has analyzed the ethical and privacy issues according to three differentdimensionsthatmayaffectthem:theoveralllearninganalyticsapproach,theparticularsolutiontolearninganalyticsadopted,andtheeducationalcontextswheretheanalyticsareapplied.Thereflectionpresentedinthisworkshowsthattheissuesassociatedwithsmaller-scaleteacher-ledlearning analytics require explicit consideration of the role of teachers as main actors in theapplicationofthelearninganalyticsprocesses.Aspectssuchasactionandimpacthaveaparticulardimension, and are closer to general ethical issues related to classroom orchestration, whereteachersplay a crucial role. In this kindof educational context, it is necessary to analyze fromanethical point of view what kinds of actions should trigger the learning analytics output, how toproceed,andwhattheimpactwillbeintermsofclassroommanagement,intervention,regulation,andassessment. Wehaveshownhowourapproachtolearninganalytics,involvingteachersfromtheverybeginninginthedesignandenactmentofthemonitoringprocesshelpstoovercomeissuesthatcanappearinthesesmallercontexts,suchascontroloftheanalytics,awareness,etcetera.Teacherparticipationin the design of the monitoring process contributes to the development of “ethics by design”(parallel to the idea of “privacy by design” [Schaar, 2010] in the application of learning analyticssolutions).Accordingtothisschema,theteacheristheonewhodefinestheeducationalpurposesoftheanalysis, reflectson theavailabledata sources, improves thevalidityof the results (byaddingnewevidencefromteachersandstudents),andisawareofthelimitationsoftheresultsobtained. Finally, the application of the approach to two educational contexts shows that reflection onlearning analytics must take into account the specific ways of working associated with differentcontexts,includinglegalaspectsthatapplytotheparticularcaseofschools,whereworkwithminorsposesspecificchallengesandpromptstheinclusionoffamiliesasnewactorsintheframework.The



cloud-based tools that arebecomingwidespread at theseeducational levels donot dealwith theinformation needs of LA andmay pose legal and ethical problems related to data ownership andvirtual identity, which are difficult to solvewhenworkingwithminors. Not only LA, but also thewider technology-enhanced learning community facesabig challengewhenprovidingappropriatetoolstotheseeducationallevels. This work is a first step towards the refinement and adaptation of ethical frameworks to thedifferent approaches that currently coexist in the learning analytics field; for example, definingdifferent itinerariesdependingontheapproachtoLA,sothatpractitionerscan focusonthemostimportantquestionsfortheirapproach.8 ACKNOWLEDGMENTS This research has been partially funded by the European Commission’s Lifelong LearningProgramme,ProjectNumber526965-LLP-1-2012-1-GR-COMENIUS-CMPandbytheSpanishProject,TIN2014-53199-C3-2-R.TheauthorswouldalsoliketothanktherestoftheGSIC/EMICGroupattheUniversityofValladolidfortheirsupportandideas,aswellasteacherDanielMantillaFernándezandthestudentsandfamilieswhoparticipatedinthestudy. REFERENCES Avouris, N., Fiotakis, G., Kahrimanis, G., Margaritis, M., & Komis, V. (2007). Beyond logging of

fingertipactions:Analysisofcollaborativelearningusingmultiplesourcesofdata.JournalofInteractiveLearningResearch,18(2),231–250.

Blackwell,C.K.,Lauricella,A.R.,Wartella,E.,Robb,M.,&Schomburg,R.(2013).Adoptionanduseoftechnology in early education: The interplay of extrinsic barriers and teacher attitudes.Computers&Education,69,310–319.http://dx.doi.org/10.1016/j.compedu.2013.07.024

Dillenbourg, P. (2013). Design for classroom orchestration.Computers & Education, 69, 485–492.http://dx.doi.org/10.1016/j.compedu.2013.04.013

Ebner,M.,&Schön,M.(2013).Whylearninganalyticsinprimaryeducationmatters!BulletinoftheTechnicalCommitteeonLearningTechnology,15(2),14–17.

Ferguson,R.(2012).Learninganalytics:Drivers,developmentsandchallenges.InternationalJournalof Technology Enhanced Learning, 4(5–6), 304–317.http://dx.doi.org/10.1504/IJTEL.2012.051816

Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework forlearninganalytics.JournalofEducationalTechnology&Society,15(3),42–57.Retrievedfromhttp://ifets.info/journals/15_3/4.pdf

Griffiths, D. (2012). The implications of analytics for teaching practice in higher education. CETISAnalytics Series, 1(10). Retrieved from http://publications.cetis.org.uk/wp-content/uploads/2013/02/Analytics-for-Teaching-Practice-Vol-1-No-10.pdf

Hylen,J.(2015,October1).LearninganalyticsintheNorwegianschoolsystem.LACEprojectarchives[Weblogpost].http://www.laceproject.eu/blog/learning-analytics-in-the-norwegian-school-system/

Kay, D., Korn, N., & Oppenheim, C. (2012). Legal, risk and ethical aspects of analytics in Higher



education.CETISAnalyticsSeries,1(6),1–30.Lim,C.P.,Zhao,Y.,Tondeur,J.,Chai,C.S.,&Tsai,C.C.(2013).Bridgingthegap:Technologytrends

anduseoftechnologyinschools.JournalofEducationalTechnology&Society,16(2),59–68.Pardo,A.,&Siemens,G.(2014).Ethicalandprivacyprinciplesforlearninganalytics.BritishJournalof

EducationalTechnology,45(3),438–450.http://dx.doi.org/10.1111/bjet.12152Rodríguez-Triana,M.J.,Martínez-Monés,A.,&Asensio-Pérez,J.(2011).Monitoringcollaborationin

flexibleandpersonallearningenvironments.Interaction,DesignandArchitecture(s)Journal,Special Issue: Evaluating Educative Experiences of Flexible and Personal LearningEnvironments,11(12),51–63.

Rodríguez-Triana,M.J.,Martínez-Monés,A.,Asensio-Pérez,J.I.,&Dimitriadis,Y.(2012).Towardsamonitoring-awaredesignprocessforCSCLscripts.InV.Herskovic,H.U.Hoppe,M.Jansen,&J. Ziegler (Eds.), Collaboration and technology (Vol. 7493, pp. 223–236). Berlin: Springer.http://dx.doi.org/10.1007/978-3-642-33284-5_21

Rodríguez-Triana,M.J.,Martínez-Monés,A.,Asensio-Pérez,J.I.,&Dimitriadis,Y.(2013).Towardsascript-aware monitoring process of computer-supported collaborative learning scenarios.International Journal of Technology Enhanced Learning, 5(2), 151–167.http://dx.doi.org/10.1504/IJTEL.2013.059082

Rodríguez-Triana,M. J.,Martínez-Monés,A.,Asensio-Pérez, J. I.,&Dimitriadis, Y. (2015). Scriptingandmonitoringmeeteachother:Aligninglearninganalyticsandlearningdesigntosupportteachers in orchestratingCSCL situations.British Journal of Educational Technology,46(2),330–343.http://dx.doi.org/10.1111/bjet.12198

Schaar, P. (2010). Privacy by design. Identity in the Information Society, 3(2), 267–274.http://dx.doi.org/10.1007/s12394-010-0055-x

Sclater,N.(2014).Codeofpracticeforlearninganalytics:Aliteraturereviewoftheethicalandlegalissues. Jisc. Retrieved from http://repository.jisc.ac.uk/5661/1/Learning_Analytics_A-_Literature_Review.pdf

Sclater, N. (2015,March 3). A taxonomy of the ethical and legal issues of learning analytics v0.1.Effective Learning Analytics. Jisc. [Web log post]. Retrieved fromhttp://analytics.jiscinvolve.org/wp/2015/03/03/a-taxonomy-of-ethical-legal-and-logistical-issues-of-learning-analytics-v1-0/

Sclater, N., & Bailey, P. (2015). Code of practice for learning analytics. Jisc. Retrieved fromhttp://www.jisc.ac.uk/guides/code-of-practice-for-learning-analytics

Slade,S.,&Prinsloo,P.(2013).Learninganalyticsethicalissuesanddilemmas.AmericanBehavioralScientist,57(10),1510–1529.http://dx.doi.org/10.1177/0002764213479366

Wise,A.F.(2014).Designingpedagogicalinterventionstosupportstudentuseoflearninganalytics.Proceedingsofthe4thInternationalConferenceonLearningAnalyticsandKnowledge,203–211.http://dx.doi.org/10.1145/2567574.2567588

learning analytics in small-scale teacher-led innovations: … · 2017-02-27 · learning analytics...

Documents