title learner dashboards a double-edged sword? students ... · 1.1 la affordances in k–12...

Title Learner dashboards a double-edged sword? Students’ sense-making of a

collaborative critical reading and learning analytics environment for fostering 21st-century literacies

Author Jennifer Pei-Ling Tan, Elizabeth Koh, Christin Jonathan and Simon Yang Source Journal of Learning Analytics, 4(1), 117–140. Published by UTS ePRESS

Copyright © 2017 The Authors This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/). Citation: Tan, J. P. L., Koh, E., Jonathan, C., Yang, S. (2107). Learner dashboards a double-edged sword? Students’ sense-making of a collaborative critical reading and learning analytics environment for fostering 21st-century literacies. Journal of Learning Analytics, 4(1), 117-140. http://dx.doi.org/10.18608/jla.2017.41.7 This document was archived with permission from the copyright owner.

http://creativecommons.org/licenses/by/4.0/

(2017).Learnerdashboardsadouble-edgedsword?Students’sense-makingofacollaborativecriticalreadingandlearninganalyticsenvironmentforfostering21stcenturyliteracies.JournalofLearningAnalytics,4(1),117–140.http://dx.doi.org/10.18608/jla.2017.41.7

ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 117

Learner Dashboards a Double-Edged Sword? Students’ Sense-Making of a Collaborative Critical Reading and Learning Analytics

Environment for Fostering 21st-Century Literacies

JenniferPei-LingTan,ElizabethKoh,ChristinJonathan,andSimonYangNationalInstituteofEducation,Singapore

[email protected]

ABSTRACT: The affordances of learning analytics (LA) dashboards and visualizations arebeing increasingly harnessed for enhancing 21st century (21C) pedagogical and learningstrategies and outcomes. However, use cases and empirical understandings of studentexperience,especiallyintheK–12schoolingsectorandinAsianeducationcontexts,remainrelativelyscarceinthefield.Ourpaperaddressesthisknowledgegapintwoways.First,wepresentafirstiterationdesignofacomputer-supportedcollaborativecriticalreadingandLAenvironment,WiREAD,andits16-weekimplementationinaSingaporehighschool.Second,weforegroundstudents’evaluativeaccountsofthebenefitsanddrawbacksassociatedwithWiREAD’s LA dashboard,which pointed to a number of potentialities and perils. Positivesincluded 1) fostering greater self-awareness, reflective, and self-regulatory learningdispositions,2)enhancinglearningmotivationandengagement,and3)nurturingconnectiveliteracy among students. The motivational value of peer-referenced LA dashboardvisualizations for stimulating healthy competition and game-like learning was identifiedalongside theperils of these serving todemoralize, pressurize, and trigger complacency inlearners.ThispaperaimstoshedlightonthepedagogicalcomplexitiesofdesigningLAthatconsiderslearnersasacriticalstakeholdergroup.

Keywords: Learning analytics, computer-supported collaborative learning, critical literacy,21stcenturycompetences,dashboards

1 INTRODUCTION

1.1 LA Affordances in K–12 Learning Contexts

The affordances of learning analytics (LA) dashboards and visualizations are being increasinglyharnessed to enhance 21st-century (21C) pedagogy, learning, and literacies. According to Verbert,Duval, Klerkx, Govaerts, and Santos (2013), having access to formative feedback from learningdashboards gives learners the opportunity for greater awareness and reflection on their learning,usingtheinsightsgainedtomodifytheirperceptionsorbehaviours.WhileLAdashboardsarestillafairly nascent educational tool, the benefits of using LA tools have been highlighted in extantliterature,withtheuseofLAdashboardsbeingassociatedwithbetterperformance(Arnold&Pistilli,2012), greater opportunities for reflection, and increased motivation in students (Verbert et al.,2013;Wise,Zhao,&Hausknecht,2014).Atthesametime,studiesindicatethattheeffectivenessofformativefeedbackislargelydependentonstudents’perceptionsofthefeedbackreceived(Stiggins,2005;Wiliam,2011), and thatnotall LAdashboardsareuseful for learning; somemayevenhave



negative effects on student learning. For example, Beheshitha, Hatala, Gašević, and Joksimović(2016) reported that the quality of students’ commentswas positively associatedwith the use ofquality-related dashboard visualizations but negatively associated with the use of class averagedashboardvisualizationsforstudentshighonself-avoidancegoals.

Onthisfront,however,amuchhigherproportionofstudiesexaminingthepedagogicalbenefitsandcomplexitiesassociatedwithLAaresetinthecontextofhighereducationandmassiveopenonlinecourses (MOOCs), as compared to research in K–12 settings. To illustrate, of approximately 790articles with LA as a keyword extracted from theWeb of Science, EbscoHost, and ScienceDirectdatabasesbetween2011andSeptember2016,30.5%focusedonhighereducationand10%relatedtoMOOCs,whileamere32papers(4%)werespecifictoK–12learningcontexts.Further,aclearlackofresearchhasbeennotedonuse-casesoflearnerdashboardsinK–12settings(Schwendimannetal., 2016). Consequently, research-informed empirical understandings of students’ personalexperienceswithLAtoolsandenvironmentsatfostering21Cliteraciesremainscarcetodate.Thisisespeciallysointhesecondaryschoolingsector,andinAsianeducationcontextsinparticular(Ochoa,Suthers,Verbert,&Duval,2014;Schwendimannetal.,2016).Thispaperaddressesthisknowledgegapby1)presentinga first iterationdesignofa computer-supportedcollaborativecritical readingand LA environment and its 16-week implementation in a Singaporehigh schoolwith 15-year-oldstudents;and2)foregroundingthesestudents’evaluativeaccountsandcollectivesense-makingtodrawoutthepedagogicalbenefitsandcomplexitiesofdesigningand implementingLAdashboardstofoster21Cliteraciesandlearningdispositions.

1.2 LA and 21C Literacies

Amarkedshift incontemporaryunderstandingsof literacyoccurred inthemid-1980swhen itwasreconceptualized as a social practice inextricably linked to its context. Since then, notable workssuch as the New Literacy Studies (Gee, 1991, 2008; Street, 2003), Multimodal Literacy (Jewitt &Kress,2003),SocialLiteracies(Street,2014),andMultiliteracies(NewLondonGroup,1996;Cope&Kalantzis,2015)havebeen influential in redirectingviewsof literacy towardsapluralensembleof“literacies” — or literate behaviours that are “increasingly multiple, multimodal and mediatedthroughnewtechnology”(Burnett,Davies,Merchant,&Rowsell,2014,p.1).Thesetheorizationsofnew literacies share a central commitment to the importance of developing in learners both theability and disposition to critically frame, evaluate, understand, make-meaning, and use multipleformsoftextinsociallygenerativeandproductivewaysacrossdiversecommunicativeandlearningcontexts.This inturnencapsulatesourconceptualizationanduseoftheterm21C literacies inthispaper.Followingthisview,criticalreadingdevelopmentistakenasanessentialcomponentofstronglanguageand literacyskills thatare fundamental toyoungpeople’sproductiveparticipation in the21Cglobalknowledgeeconomy.Despite its importance,however, ithasbeencommonlyobservedbyteachersinSingaporeandinternationallythatstudentsstrugglewithrelatingandrespondingtoEnglish language (EL) texts on a personal level, thereby failing to engage deeply and critically inreadingtasks(Garcia,Mirra,Morrell,Martinez,&Scorza,2015).This,inturn,hasbeenattributedtotwokeyfactors.



First, texts used in EL lessons are still predominantly print-based (Cope & Kalantzis, 2015), eventhough today’s youth now interact largelywith both printand digitalmultimodal texts— that is,textsthatincorporatetwoormorecommunicationmodesformeaning-making,suchasverbalandwritten language,visual imagery,gesture,andsound (Jewitt&Kress,2003;Walsh,2010).Further,despite their continual engagement with rich and fluid social media platforms beyond school,students have limited opportunities in the conventional EL curriculum and pedagogical setup toextendtheirreadingpracticesandinteractionsinsocially,cognitively,andtechnologicallyengagingways beyond the stipulated class time (Tan & McWilliam, 2009). A second and more pertinentreason is that students currently have very limited access to meaningful and timely formativefeedbackonvariousdimensionsof theirpersonal readingengagementanddevelopmentprogress(Davison, 2013), thereby significantly constraining their capacity to self-evaluate and positivelymodifytheirlearningbehaviours,eveniftheysodesired.

Untiltheseissuesareeffectivelyaddressed,students’lowlevelsofELengagementandweakcriticalreading skills are likely to persist, with adverse effects on their language proficiencies, and 21Ccapacities — critical thinking, collaboration, and communication. This challenge serves as theimpetus for our design, implementation, and evaluation ofWiREAD, a web-based collaborativecritical reading and LAenvironment aimedat fostering21C literacies— that is, deepening criticalreadingengagementlevels,promotingself-regulatedandcollaborativeknowledgeconstructionasasocially generative practice in the EL domain among Singapore Secondary 3 (Grade 9) students,duringandbeyondformalclasstime.

2 WIREAD: A COLLABORATIVE CRITICAL READING AND LA ENVIRONMENT

WiREADwasdesignedwiththeprimaryobjectiveofmotivatingandscaffoldingstudentstodevelopricher dialogue and quality interactions with peers around multimodal texts, thereby deepeningtheir personal connection to and appreciation of collaborative and critical reading as a highlyrelevant,generative,andmeaningfulsocialpractice.Toachievethis,thetechno-pedagogicaldesignofWiREADfocusedontwokey learningaffordances:onlinepeer interactionsaroundreading,andtheLAdashboard.

2.1 Online Social Reading and Discussion Space

AnonlinesocialreadinganddiscussionspacewasdevelopedinWiREAD.Thislearningaffordanceisunderpinned by Vygotskian socio-constructivist theories, and informed by a multiliteraciespedagogical framework that foregrounds four key dimensions of effective contemporary literacyenculturation in learners: 1) situated practice, 2) overt instruction, 3) critical framing, and 4)transformedpractice (Cope&Kalantzis, 2015; Tan&McWilliam, 2009). This online social readinganddiscussionspacewasdesignedforstudentstosimultaneouslyview/readmultimodaltextsandcollaboratively critique and discuss these texts with their peers, using a suite of intentionallydesignedmicro-pedagogicalscaffoldingscriptsandpopovers(Figure1).

The micro-level of pedagogical scaffolding scripts comprised 7 critical lenses (Message, Purpose,Audience, Assumption, Viewpoint, Inference, Impact) and 5 critical talk types (Ideate: I think that…;Justify: I thinksobecause…;Validate: Iagree…;Challenge: Idisagree…;Clarify: Ineedtoask…).Paul



andElder’s(2001)“wheelofcriticalreasoning”andourownworkondialogicindicatorsofcollectivecreativity and criticality (Tan, Caleon, Jonathan, & Koh, 2014) informed the design of these criticallenses and talk types. Together, these served as a meta-cognitive schema for guiding students’collaborative critique of texts on WiREAD, in that students are required to tag each of theircomments/replieswithonecriticallensandonecriticaltalktype.Thisisinlinewithpastresearchthathasadvocatedsuchfunctionalityforpersonalizedagenticlearning(Beaudoin&Winne,2009)andwasfoundtohelpstudentsdevelopperspective-takingandmeaning-makingcapabilities(Eryilmaz,vanderPol,Ryan,Clark,&Mary,2013).Furthermore,eachcritical lensandcriticaltalktypetagcontaineda“popover” that provided studentswith question prompts and sentence starters. These served as aconstantreferentialresourceremindingstudentswhateachtagmeant,andhowthesecouldbeusedtocritiquetextsmoredeeply.

Figure1:Collaborativecriticalreadinganddiscussionoftextsusingpedagogicalscaffoldingscriptsofcriticallensesandcollaborativetalktypes.

A new multimodal text (incorporating a blend of written text, visual imagery including gesturalmodes,andsometimesvideo)thatcentredonapertinentsocial,moral,and/orethicaldilemmawasuploaded eachweek over a period of 16weeks across two school semesters (Figure 2). In theseweeklyWiREADsessions,studentshadone30-minuteperiodduringtheirformalELcurriculumtimeinschooltoread,comment,andreplytootherclassmates’postsonthetexts.Thisweekly30-minute

Expanded view



periodconstitutedone-thirdof theweeklySecondary3ELcurriculumtime(3x35minuteperiodsper week for 16 weeks), and findings reported and discussed in subsequent sections should beinterpretedinlightofthislevelofexposurethatstudents’hadtoWiREADovertheimplementationperiod.Thiswas themaximum formal curriculum time that theparticipating school leaderandELteachers collectively decided they were able to devote to this curriculum innovation withoutcompromising other essential academic demands of the EL subject domain. Students were alsoencouragedtouseWiREADintheirpersonaltimebeyondformalELlessons.

Figure2:Multimodaltextsco-developedbyteachersandresearchers(examplesonly).

Architecturally,WiREAD is built on WordPress, an open-source PHP website creation tool, withadditional plugins and in-house programming codes to customize the functionalities for ourpurposes.OnemajorpluginwasBuddyPress,whichenhancedthesociallearningaspectwhereuserscan join singular or multiple groups, and utilize interactive features including an activities feed,personalmessaging,andafriendshipsystem.AsshowninFigure3,alldataarestored inaMySQLdatabase, with structural modifications made to capture additional data required for the LAdashboard component of the learning environment. This is further explicated in the followingsection.

Figure3:WiREADsystemarchitecture.



2.2 LA Dashboard

TheLAaffordanceofWiREADwasdesignedwith theaspirationofproviding rich,meaningful,andtimely formative feedback to studentsand teachers throughout the term tohelpmonitor readingengagement and progress, and modify learning strategies and pedagogical practices to improveoutcomes. We focus here on the student LA dashboard, which consisted of four componentsdesigned to visualize a rangeofdispositional, discourse, and social networkanalytics (Ferguson&BuckinghamShum,2012)alongsideELachievementdata.WeelaborateoneachofthefourWiREADLAcomponentsbelow:

• My WiREAD Critique and Discussion Profile (Figure 4): discourse-related learning data onstudents’ online reading engagement based on their frequency and length of comments andreplies,aswellastheirusageofdifferentcriticallensesandcriticaltalktypes(describedearlierinSection2.1)acrosstexts.

Figure4:MyWiREADCritiqueandDiscussionProfile.



• My Learning Attitudes and 21C Skills Profile (Figure 5): dispositional learning data fromstudentself-reportquestionnairesadministeredatthestartandendofeachschoolsemester,using measures adapted from pre-validated scales of productive 21C learning dispositions(Tan,2009;Tan&Nie,2015).These included: fivedimensionsof21Cskills (critical thinking,creativity, curiosity, collaboration,open-mindedness); threedimensionsofattitudes towardslearning EL (self-efficacy, task value, and engagement); five dimensions of student–teacherrelatedness (student–teacher communication, trust, alienation, autonomy, and competencesupport fromtheteacher);andfourdimensionsof learninggoalsandmindset (masteryandperformance goal orientations; deep and surface learning). Examples of items for eachdimension are shown in Appendix A. The students used a 7-point Likert response scale(1=strongly disagree to 7=strongly agree). The Cronbach alpha reliabilities associated witheachdimensionweresatisfactory(.71–.92).

Figure5:MyLearningAttitudesand21CSkillsProfile.



• MyWiREADSocialLearningNetworkMap(Figure6):sociogramsreflectingstudents’positionandinfluencewithintheWiREADlearningnetwork(tiesbasedonstudents’onlinediscussionof textswithothers)and theclass’ learningnetwork (tiesbasedonstudents’nominationofwhoandhowoftentheyapproachedclassmatestodiscusstextscritically).

Figure6:MyWiREADSocialLearningNetworkMap.

• MyReadingAchievement(Figure7):ELachievementdataonstudents’readinggradesonschool-basedassessmentsthroughoutthetermcapturedinteachers’centralizedinformationportal.

Figure7:MyReadingAchievement.

ThedesignofthesefourLAdashboardcomponentswereinturndevelopedthrough1)apilottermof iterativedesign-basedresearchprocess involvingcognitive labsandfocusgroupswithaclassof40studentsandoneELHeadofDepartment,and2) informedbyextantgraphicalvisualizationsof



dashboardssuchastheStudentActivityMeter(Govaerts,Verbert,Duval,&Pardo,2012)andothers(Corrin&deBarba,2014;Wiseetal.,2014).

Through this two-pronged pilot design and testing process, an intentional decision wasmade onpedagogicalandresearchgroundstodisplaytheclassaveragedataasaformofsocialreferenceforstudents. This was, first, a practical decision urged by teacher-collaborators who leaned towardstheir professional experience that such peer/norm-referenced feedback had been observed to beuseful instimulatingstudents’positivechangesinlearningbehaviours.Second,thiswasaresearchintentiontogeneratecontextuallynuancedempiricaldatatoaddressanextantliteraturegapgiventhe inconclusive findings in the tertiary education sector, alongwith a definitive lack of empiricalstudies in thesecondaryschool sectoronwhether suchpeer/norm-referencedvisualizationswerehelpfuland/orharmfulforstudentlearningandinwhatways(Corrin&deBarba,2014;Wiseetal.,2014). The data indicators and corpuses for the LA dashboard visualizationswere retrieved usingPHP functions integrated into the WordPress framework. In this process, data was checked andsorted. Application algorithms were applied to the data, with processed data parsed to thevisualization handler where the final data is presented to the end user in interactive visualizedmodels.

3 METHODS

To develop more nuanced insights on the design and learning affordances of LA for optimizinglearning, we ask, “How do students make sense of the benefits and drawbacks associated withWiREAD’sLAdashboardcomponentsandvisualizations”?Toaddressthisquestion,wedrewonasubsetofdatageneratedfromourlargerdesign-basedquasi-experimental study that evaluated the impact of WiREAD’s collaborative critical reading and LAdashboard affordances on student learning outcomes (3WiREAD classes,N=116) as compared to acontrolgroup(3classes,N=92)usingacombinationofpre/post-testsandself-reportedquestionnaires,and qualitative feedback forms and focus groups conducted at the start and end of the 16-weekinnovation term. The Secondary 3 (Grade 9, 15-year-old) student participants were generallycomparable in terms of academic achievement (mid-upper academic ability track) and gendercomposition (52.9% female, 47.1% male). Given this paper’s focus on understanding studentperceptionsofWiREAD’sLAdashboard,wedrawspecificallyondatageneratedfromWiREADclasses(N=116 students) and report on findings of students’ evaluative accounts of the LA dashboardcomponents.Theseweregleanedfromacombinationofself-reportedquestionnairescalesmeasuringstudentperceptionsoftheLAdashboard’seaseofuseandusefulness,andtheirqualitative(textual)accountsexpressedduringstudentfocusgroupsandinopen-textfeedbackevaluationforms.In the questionnaire, studentswere asked to report their perceptions of the LA dashboard on twoscalesadapted fromTan (2009):1)perceivedeaseofuse (4 items)and2)perceivedusefulness (15items measuring four key productive learning dimensions: socialization, expression of identity andopinions,developmentof21Cskillsanddispositions,andacademiclearningandperformance),andanewscaleonperceivedhelpfulnessforlearningandgrowth(fouritems).Examplesofitemsforeachof



these scales and their dimensions are provided in Table 1, along with their Cronbach’s alpha (α)reliabilities,whichwerefoundtobesatisfactory(.79–.91).Table1:ExamplesofItemsofPerceivedEaseofUse,Usefulness,andHelpfulnessScales

Scale/Dimensions No.ofItems

LikertScale α ExampleofItem

Perceivedeaseofuse 4 1=stronglydisagreeto7=stronglyagree

.86 WiREADiseasytonavigate.Perceivedusefulness-Socialization-Expressidentity&opinions-Develop21Cskills&dispositions-Academiclearning&performance

3453

.91.86.86

.79

WiREADcanhelpme...…feelmoreconnectedtomyclassmates…learntovoiceandjustifymyopinionsmore…developcreativethinkingskills…improvemyperformanceinELtests

Perceivedhelpfulness 4 1=nothelpfulatall

to7=extremely

helpful

.81 HowhelpfularethefollowingWiREADfeaturesforyourlearningandgrowth?1. MyAchievementData2. MyLearningAttitudes&21CSkills3. MyWiREADDiscussionData4.MySocialLearningNetworkProfilemaps

Inaddition,studentfocusgroupswereconductedwith30students(10fromeachofthethreeWiREADclasses)whowerepurposefullyselectedtoreflectdiverseuserswitharangeof lowtohighlevelsofWiREADusage,readingachievement,andvocalityinELclass.TheremainingstudentsintheWiREADclasses who were not selected to participate in the student focus groups (N=86) completed aqualitative feedback formon theaspectsofWiREAD they found tobeuseful orotherwise for theirlearningandwhy.For thispaper,ouranalysis focusesonstudents’ textualaccountsgeneratedfromtheir responses to the following questions asked in the student focus groups and the qualitativefeedbackforms:

• Whichofthelearningdashboarddatadidyoufindusefulforyourlearning,andwhy?• Whichofthelearningdashboarddatadidyoufindnotusefulforyourlearning,andwhy?• WhatimprovementsdoyousuggestforMyLearningData/Dashboard?

The focus group discussions were audio-recorded and transcribed verbatim. Data was triangulatedfromthetranscriptsandthequalitativefeedbackformsandcompiled intocategoriestoaddresstheresearch question. Two of the authors proceeded to identify patterns from the data and draw outlargerthemesaccordingtothematicanalyticstrategiessuchaspatternmatchingadvocatedbyMilesandHuberman(1994)andYin(2009).

4 RESULTS AND DISCUSSION

4.1 Students Converged on LA Dashboard as Benefitting their Learning

Onthewhole,studentsconvergedontheWiREADLAdashboardanditsfourconstitutiveelementsasbearingmultiplepositivebenefitsthatenhancetheirlearningintheEnglishlanguageandliteracy



domainaswellasbroader21Cliteracies.Weelaborateonthesefindingsinthefollowingsections,drawing from students’ self-reported questionnaire data and their qualitative accounts of the LAdashboardcomponents.

Basedontheirquestionnaireresponses,studentsfoundtheLAdashboardeasytouse,navigate,andunderstand(M=5.54,SD=1.02,Figure8).Theywerealsogenerallypositiveabout1)eachdashboardcomponent’s helpfulness to their overall learning and growth (Figure 9), and 2) the overalldashboard’s usefulness for enhancing key dimensions of productive 21C learning— socialization,expression of identity and opinions, development of 21C skills and dispositions, and academiclearningandperformance(Oblinger&Oblinger,2005;Tan,2009;Turvey,2006),asshowninFigure10.

Figure8:WiREADeasytouse,navigate,andunderstand?

Figure9:Dashboardcomponentshelpfulforlearning/growth?



In terms of productive learning dimensions, the dashboard was perceived to be most useful fornurturingtheirexpressionofidentityandopinions(M=4.95,SD=1.37),followedbydevelopingtheir21C skills & dispositions (M=4.65, SD=1.24), then academic learning and performance (M=4.6,SD=1.28).StudentsperceivedtheLAdashboardtobeleastusefulforbolsteringtheirsocializationinschool(e.g.,connectednesstoclassmates,expansionoflearningnetwork,M=4.46,SD=1.52)(Figure10).

Figure10:Dashboardusefulforenhancingkeydimensionsofproductive21Clearning?

For deeper insights into students’ collective sense-making and “experienced realities” of the LAdashboardanditsconstitutiveelementsasbenefittingtheirlearning,wenowturntheanalyticlensto students’ qualitative accounts. Our thematic analysis foregrounded three key learning benefitsthatstudentsattributedtoWiREAD’sLAdashboard,particularly in termsof fosteringanumberofimportant 21C literacies. These are 1) fostering greater self-awareness, reflective, and self-regulatory learning dispositions, 2) enhancing learning motivation and engagement, and 3)cultivatingconnectiveliteracyamongstudents.

4.1.1 Fostering greater self-awareness, reflective and self-regulatory learning dispositions StudentsrecurrentlyarticulatedtheusefulnessoftheLAdashboardintermsofitsvalueformakingvisibletheirlearningprogress,dispositions,andbehaviours,whichtheyimpliedwerelargelyvague,even invisible to them in their everyday schooling endeavors. Going further, students repeatedlydescribed the LA dashboard and its various components as most informative in terms fosteringgreater levels of self-awareness and reflection of one’s learning profile and progress, therebypromptingintentionalgoal-settingandself-regulatoryadaptivelearningstrategiesontheirpart.Interestingly,studentsfrequentlyconvergedontheLearningAttitudesand21CSkillsProfileasthemost“eye-opening”inhelpingthem“discover,”“realize,”“understand,”and“findoutstrengthsandweaknesses”thattheydidnotpreviouslyknowaboutthemselves:

“I found thatMy Learning Attitudes and 21C Skills were themost useful. This dashboard



allowedmetorealizewhattypeofattitudeIhadtowardslearningEnglishandthewaythatIwastryingtoachievemyELgoals.I'vecometorealizethatmyattitudetowardsELwasnotthebestasIonlylearntforthesakeofdoingwellinexamsnotforthesakeofunderstandingthe language. Thus looking at it, I am trying to change my learning habits and attitudetowardsEnglishinabetterandmorepositivedirection”(St1,3R7).

The learning benefit of this dispositional analytic was echoed in many other similar studentaccounts:

“IfounditveryinterestingandithelpedmeknowmoreabouthowIlearn,thushelpingmeadjustthewayIlearntandstudied.Italsohelpedmeunderstand21stcenturyskillsandhowmuch of each I displayed. Knowing my learning attitudes also helped me to understandwhereIamlackingandwhereIshouldworkon.”(St17,3R6)

Theselearneraccountsreinforcetheneedforschoolstoplacemoreovertemphasisondevelopingstudents’ 21C literacies and dispositions by appropriating the affordances of pedagogicallymeaningful andwell-designed LA, not only because these serve as strong predictors of academicoutcomes (Tempelaar, Rienties,&Giesbers, 2015), but evenmore so of longer-termemploymentandlifeoutcomes(Levin,2012).

In a similar vein, students found theWiREAD Critique and Discussion Profile to be beneficial inraisingtheirself-awarenessandself-reflection,therebypromptingintentionalgoal-settingandself-directedadaptivelearningstrategiestoexpandtheircriticalliteracyfluency:

“I find that the Learning Attitudes and theWiREADDiscussion Data is useful. It helpsmezoom in on where I am lacking and how I can further improve on the missing skills andtechnique Iamnotapplying. Itallowsmeto learnmoreonhowtoachieveanall-roundedanswerinmyfutureanswers.Ithassofarproveneffective!”(St14,3R6)

“I foundMyWiREADDiscussionDatausefulas it showsmemystrengthsandweaknesses,thissection let[s]megaugemyability toanswercomprehensionquestions. ItalsoallowedmetoestimateandknowhowmuchIamlackingandplanoutwhatIneedtodoinordertoimprove.”(St3,3R6).Anotherclassmateelaborated,“LearningDashboardshowsthegraphsonthecriticallensthatoneoftenuses.Thiswillenableustorealizewhatwehavenottriedbefore andwhich critical lenswe should try outmore. This improves the exposure to thedifferentcriticallensandthemorewepractiseononecriticallens,themoreweareabletoapplyit.”(St26,3R6)

Consistentaccountswerealsogivenby studentsof theReadingAchievementdashboard and theWiREADSocialLearningNetworkMap fostering their reflectiveandself-regulatory resourcesasanaturallyoccurringelementoftheirlearningprocess:

“[TheReadingAchievementdashboard]wasusefulformylearningasitshowedmyprogressfor thevarious lessonsandallowedmetoevaluateandunderstandmy learningprogress.”(St14,3R1).Anotherpeerintheclasscontinued,“IalsofoundtheELachievementdatamostuseful…asIwasabletogaugehowmuchprogressforELIhavemadeaswellashowIfairagainstotherstudents…throughthis,IknowwhatIcanworkoninordertoimprove.”(St19,3R1)



“MySocialLearningNetworkProfile[wasmostuseful].Itshowswhoyouhaverespondedtoandviceversa.Thethickerthe linesmeansthemoreoftendiscussionoccursbetweentwopeople and those who have greatly contributed to discussions are placed nearer to thecentre. I find it useful as it showsmyprogress andwhich direction I should be headed inordertogetpositionednearertothecentre.ItalsoshowsmewhoIshouldhavediscussionswith…[that]willbemoreinsightfulandfruitful.”(St2,3R6)

4.1.2 Enhancing learning motivation and engagement On a related note, frequently heard among the participant cohort were students’ attributions ofstrong links between greater learner self-awareness and reflection as precursors of enhancedlearningmotivationandengagement—cognitive,affective,andbehavioural(Wang&Eccles,2012).Forexample,St8shared:“Itwasuseful.ItallowedmetoknowwhatI’matnow,sothatIcanworkhardertoimprove.Soyah,IworkedslightlyharderandIgotbetterformy[SemesterAssessment]2”(3R7-Line_217). Similar accountswere articulated in other classes: “MyELAchievementData andWiREAD discussion data was the most useful. It shows where I stand in class, whether I’munderperforming or not.Hence, itmotivatesme towork harder and try new things, so that I cancatchupwithmyclassmates.”(St8,3R6)

Inparticular,studentsdescribedtheWiREADSocialLearningNetworkMapthatvisualizeddynamicrelational learning activity data as being especially engaging and motivating. Students frequentlyvoicedcommentssuchasthefollowing:

“showsmewhereIstand…motivatesmetoworkharder,”“…madememoremotivatedtocommentonother’sanswers” (St16,3R7),“…itmakesmemoremotivatedtocommentsothatmy[socialnetwork]dotcanbebiggerandbrighter,andIwillknowwhotolooktoforhelp” (St16, 3R7), “…motivate me to help or get help from my friends,” “…serves as aremindertowritecommentsontextswehavenotdone.Italsoencouragesmetobemoreactiveas Ioftenvisit thissectiontoseemyprogressandeffort” (St8,3R7),“…itshowsmetheparticipationleveloftheclassandseehowactiveyouareascomparedtotheclass.Iwillknowwhere Istand intheclassand if it’sbelowtheaverage, likebeingattheedgeofthenetwork,Iwillbemoredeterminedtoanswermorequestionsandcommentonmyfriend’sanswers, increasing my level of participation for the text.” (St20, 3R6), and “…it’s like toencourageyourselftoextend…thecircleofpeoplewhoyoudiscussEnglishtextwith…soit’slike,toseeyourbubblegrowandit’slike,itisanincentive…likediscussmorewiththewholeclass,insteadofjustyourfriends”(St23R1-Line306and312)

These student-user accounts concur with much of LA research on the potential of socialtransparency and “visible learning” for promoting self-regulated learning (Lockyer, Heathcote, &Dawson,2013)andalso that learnerautonomy,competence,and relatednessarevital to intrinsicmotivation and sustained behavioural change (Wise, 2014), following the research on self-determinationtheorybyDeciandRyan(2011).

4.1.3 Cultivating “connective literacy” — a nascent disposition among students Instudents’collectivesense-making,alsonoteworthywastheirrecognitionthattheLAdashboard,especially its social network analytics, bore much value for nurturing a disposition we termreciprocalsociallearning,thatis,acapacitytoshiftfocusawayfromself-interestandself-sufficiency



toengageinproductivehelp-givingandhelp-seekinglearningbehaviours.Instudents’words:

“Thespiderweb, someofournames, thebubbleswerebigger, right?So it showedus likewhocanwe find like ifweeverneedanyhelp inEnglish.And someof thepeople,whosebubblesarelikesmaller,theyareoutofthewebthing,sowecanlikewecanjustaskthemwhethertheyneedanyhelp.”(3R1,Line270–285)

“…itmakesmemoremotivatedtocommentsothatmy[socialnetwork]dotcanbebiggerandbrighter,andIwillknowwhotolooktoforhelp…motivatemetohelporgethelpfrommyfriends.”(St16,3R7)

Thisideaofreciprocalsociallearningbringsustothelargerconceptofconnectiveliteracywhichweconceptualizeanddefineasthecapacitytoseelearningasanadaptiveandfluidnetworkedactivity,wheresocialpositioningisseenasconstitutingaformoflearningcapitalrequiringactivenurturingand reciprocal transactivity with significant others. In other words, connectively literate studentsrecognizetheimportanceof“seeing”theirindividualpositioningwithinthelargerclassnetwork,ofknowingwhomtheywere“connectedwith”and“stay[ing]connected”inwaysthatwereprofitabletolearning,suchas:

1. To“findoutthepeoplethat[they]canactuallyapproachforEnglish,”to“knowwho[they]need to interactwithmore,” to seekhelp from“prominentothers” so as “to improveandlearnfromthem”

2. To“diversify”and“expand thecircleofpeoplewhom [they]discussEnglish textswith… totakepartinmoreconversationswiththewholeclass,insteadofjustthepeople[they]knowbetterasfriends,”thereby“mak[ingthem]thinkmore”or“forcingthemtothinkdeeper”

3. Toshowconsiderationto“othersatthesidewithhardlyanyconnection…[to]trytoseetheircommentsmore,andreplytotheircomments”

Onamoresoberingnote,however,wenoted low levelsof this“connective literacy”amongst thestudent participants. The views articulated abovewere noted to be relatively nascent and yet toemergeamongmanystudentparticipants.Instead,moststudentswereheardtoprivilegeindividuallearning outcomes and skill acquisition consistently asmuchmore important than social learningconnections:

“Tome, it doesn’tmatterwho I’m connected to. As long as I can approach the textwithdifferentcriticallenses,itisusefulandhelpful.MySocialLearningNetworkProfiledoesnothelpmelearnanything”(St8,3R1).Othersreferredtothesociallearningnetworkprofileasa“redundantsystem,”because“allitreallydidwasshowwhoworkedwithwhomore.Ithadnorealsignificantpurposetohelpwithmylearning”(St29,3R1),withyetothersstating,“allI’minterestedinisthecontentofthecomments”(St1,3R6).

Anotherstudentsimilarlyadded:

“I foundthepartwhichshowedme‘MySocialLearningNetworkProfile’notuseful.This isbecause I do not believe that there is a need to know about other people’s progress or



whether or not they contribute to replying to others’ comments or whether they askquestions,aslongasIcompletethosetasksmyself.”(St3,3R6)

Tomanyofthesestudents,socialnetworkshavelittlebearingforlearningandgrowth,withnetworkcentralityorprominencebeingtrivializedaspromotingasuperfluous“socialbutterfly”syndrome,attheexpenseofmore“seriouslearners…thosewhotrulyliketocommentandlearn”(St10,3R6).Theinherently low levelsof “connective literacy” amongst adolescent learners has implications for notonly LA designers and pedagogues, but for awider educational community committed to helpingstudentsbecomeproductiveandengagedcitizensbeyondschool.Thoughweacknowledgetheneedfor a critical evaluation ofWiREAD’s social learning network dashboard design and pedagogicalvalue,thesestudents’accountsreflectanemergentunderstandingofthepowerofsocialnetworksandhowtheseextensively influence ideas,emotions,behaviours, learning,andmore(Christakis&Fowler,2009).Thisinturnbringstoourattentionanoften-overlookedimperative—thateducatorsofyoung leanerstodayneedtobemuchmore intentionalandpurposeful ineffortstodevelopanempiricallyinformedappreciationforthepowerofadaptiveandreciprocalsocialnetworkedcapital— as a form of new fundamental literacy essential to successful learning, living, and earning incontemporarysocieties.WiREADprovidesonesuchopportunityforbuildingtheseliteracies.

4.2 Peer-Referenced Visualizations: A Double Edged Sword

Students’ qualitative accounts of the benefits gained from the LA dashboard, however, drewopposingcounterpointsfromotherpeers,thusprovidinginsightsintotheambivalencereflectedinresponsesontheusefulnessofthedashboardfortheir learningandgrowth.Wehighlightonekeyproblematic here — that of the ironic nature of peer/norm-referenced versus self/criterion-referencedlearningvisualizations.

4.2.1 Motivational Value of Peer-Referenced Visualizations: Healthy Competition, Game-Like Learning

Students were polarized in their views of the norm-referenced visual analytics as indicators ofacademicand social standingamongpeers, and thusas amotivatingmechanism for learning. Formany,knowing“wheretheystandinclass,” if theyare“belowtheaverage”or“attheedgeofthenetwork” made them more “determined… to increase [their] level of participation,” to “workharder,”“bemoreactive,”and“improve”their learningbehavioursonline.Further,studentsoftenalludedtotheinherent“fun”and“interesting”natureofthevisualizationsasstimulatingaformofgame-like learning (Gee, 2008) through “healthy peer pressure” and informal “competition” thatdrivesthemtoengagewithlearninginmoresubstantiveways:“thespiderwebwouldlike,forme,itlooks likeaminigame,sothat it’saracetogettothecenterandtogetasbigasyoucan” (St12,3R7).Higher-performingstudentsintheclassesevencalledforamoreformalized“rankingsystem”or“halloffame,”assertingthatthis“leadstoadvancementandconstantusageofWiREAD”(St25,3R7).Astudentexplained:“…competitivenessisawayforyoutoimproveactually…likeifyou…youwanttoalways likebeat thepersonthat’s thebestone, right…sothatyoucanbethebest” (3R1-Line_752). The network visualization also allowed students to gauge their influence in the classdiscussionand“motivated”themto“becomethemostinfluentialamong”theclass.



4.2.2 Perils of Peer-Referenced Visualizations: Demoralizing, Pressurizing, and Triggering Complacency

Therewere,however,opposingviews,andsomepointedtotheadverseaffectiveimpactof“clearlyseeing”oneself in relation toothers in the class, especially if one is “at thebottom” (MyReadingAchievement),or“attheoutside”(MySocialLearningNetworkMap).Afewstudentsfeltthatitwasdemoralizing, inappropriate, and even discriminating to compare individual data with the classnorm:

“Itwasdemoralizing…sometimes.Youlook likeyouarebelowaverageonthechart…thentheclassaverageissohigh,andthemaximummarkyouaresupposedtogetissohigh,thenyouarebelowaverage…itwasdemoralizing…depressingtoseethatclassmatesarebetteratcommentingorthinkingskills.”(St5,3R7)

“…I do not like to compare my data with other people’s data. This might cause somestudents to feel demoralizedor depressedwhen they see that their classmates arebetterthanthematcommentingorinthinkingskills.”(St21,3R1)

“Itshows…ifoneisnotinthemiddleofweb,itmightseemasifhe/sheisoutcastedeventhoughhe/shecouldmerelybeashyindividualwhoisnotvocalenoughtoexpresshis/herthoughts.Thiscanbeseenasatypeofdiscriminationifthepersonisoffended.ThosewhoareoutsideofthetangleofwebsandlinksisapparentfromtheNetworkProfile,withoutahintofsubtletytoshowhowfarandaparttheyarefromthesocialcircle.”(St6,3R1)

Afewalsofeltthedashboardprovidedadditionalunhealthypressureforthem:

“Afterseveralclasses,Ihavefoundoutthatworkingtoabrighterandlargerspothasturnedintoacompetition…thisprovidesaddedstress thatwedonotneed,aspeoplehavetheirownpaceof learningandmaybeslower.Theslowpaceof learningmaybe laughedatbyclassmatesandthusitisnotasusefulforlearningasitmayemotionallyaffectothers.”(St14,3R6)

Besidestheadditionalstressofhavingtokeepupwiththeaverage,otherstudentssharedhowtheywouldjustputinminimalefforttohittheaverage,andnotgobeyond.Astudentdisclosedhowhestoppedhimselffromoverlyparticipatingwhenhesawthedashboard:“…becauseafterlookingatit,I see that I comment too much… and then I started to comment less” (3R1-Line_708). Anotherstudentcandidlyshared:

“Let’ssay,you justputtheaverage…thenpeoplewill just thinkthat… Iwill justdooneortwo… cause that’s the average… so thatwouldmean that the studentwould notwant toexploremore.”(3R1-Line_659)

Studentsholdingthisviewoftenassertedthatlearningvisualizationsaremosthelpfulwhentheyarecriterion-basedandself-referenced, rather thannorm-referencedtopeers:“seriously, I thinkwhatmattersmostisactuallyyourselfinsteadofotherpeople…itactuallyhassomepsychologyeffect.Youshouldn’tcomparewithanybodyelsecauseeventhoughthisiscollaborativelearning,it’sreallymoreof self-learning in the end” (St3, 3R1); and “the purposeof comparingwith others is tomake youimprove to so-calledmatch other people’s level, but comparing with yourself would, can actuallyhavethesameeffectcauseyouaretryingtoimprovebasedonyourpast.Comparingwithyourself



willbemoresuitable. If like forexample, theweekbeforeyoucommented less, thenyouthink like‘Ohyes,Iactuallycommentedmore[thisweek],Ifeelbetter,’insteadofcomparingwithotherslike‘What?I’mlast,whatisthis?Notfair’thenyoucry”(St8,3R1).

Thesepolarizingviewsspeaktotheproblematicnatureofpeer-andnorm-referencedvisualizations,whichconcurwithfindingsfromasmallnumberofexistingstudiesinthefieldofLA.InWiseetal.’s(2014)evaluationof theLAvisualizationsused in theE-Listening researchprogram, they reportedthat although many students appreciated norm-referenced visualizations, some students foundthemdiscouragingandstressful,expressingapreferenceforcriterion-basedvisualizations.Althoughthiswas in thecontextofhighereducation, the findingsaresimilar toour findings fromtheK–12context. In addition, Corrin and de Barba (2014) reported that the use of class average data caninduce a false sense of progress in students, with students being content with above-averageperformance even if it falls short of personal or course outcome targets. On a related note, theresearchconductedbyBeheshithaetal.(2016)indicatedthatindividualstudents’achievementgoalorientationsareassociatedwiththeirperceptionsofpeer-andself-referencedLAvisualizations,buttheauthorsacknowledgedthatmore in-depthresearchwasneededonthisfrontsuchthatLAcanbedesignedinwaysthatareadaptivetodiverselearnerneedsandmotivationalprofiles.

The impact of norm-referenced and criterion-referenced feedback has its theoretical roots inassessment literature (Shute,2008).Paststudies indicate thatwhenpoor-performingstudentsarepresentedwithnorm-referencedfeedback,theytendtobecomelessmotivatedandhavelowerself-efficacy,ascribingtheirperformancetoinnateabilityratherthaneffort(Chan&Lam,2010;Kluger&DeNisi,1996;Shute,2008).Students’polarizingviewsonpeer-andnorm-referencedvisualizationsalso foreground recent positive psychology understandings that emphasize “personal best”achievementgoalsasahighlysalientpredictorof students’academicmotivationandengagement(Martin&Elliot,2015).Thisperspectiveon“personalbest”achievementgoalsisupheldbystudiesshowing that the use of criterion-referenced feedback is associated with students ascribing theirperformancetoeffort(inturnreflectingamastery-orientation)andexpectingimprovementsintheirperformance (McColskey& Leary, 1985). Students’ viewson the shortcomings of peer-referencedvisualizations reinforce theuseof intrinsicmotivationmechanisms in thedesignof LA to foster amastery-orientedmindset thathasbeen shown to result inmoreadaptiveoutcomes relative toaperformance-orientedmindset(Dweck,2012).

5 CONCLUSION

Inthispaper,studentaccountsoftheLAdashboardrevealedpositivebenefitstolearningintermsoffostering greater self-awareness and self-regulatory learning dispositions, enhancing learningmotivationandengagement,andnurturingconnective literacyamongstudents.Atthesametime,the “double-edged sword” nature of peer-referenced visualizations for stimulating competition,causing undue (felt) pressure, and triggering complacency in learners was foregrounded. Thesestudent perceptions arguably reveal different expectations of the EL activities and the techno-pedagogical design of WiREAD. Wise, Vytasek, Hausknecht, and Zhao (2016) highlighted thepedagogical importanceofhelpingstudentsbalancedifferentreferencepoints (e.g.,peeraverage)and gain a better understanding of the values and limits of LA visualizations. In that regard,



WiREAD’s LA dashboard and pedagogical design could be buttressed with more scaffolding andprompts to help students engage in meta-cognitive self-regulatory learning behaviours in moremeaningfulandproductiveways.

Toconclude,wehighlightsomelimitationsandwaysforward.First,WiREADisonlyinitsfirstdesigniterationandweareconsciousoftheLAdashboard’srudimentaryvisualizations.Thereisroomforfurther enhancements to its user interface and aesthetic features. Second, we are working toincorporatericherandmoreproximalindicatorsandanalytics,andinparticularautomatedsemanticanddiscourseanalytics (BuckinghamShum&Ferguson,2012;Roséet al., 2008;Rosé&Ferschke,2016) that can better assess the quality of critical reading and thinking reflected in students’discursivepractices.Relyingonstudents’self-taggingofcriticallensesandcriticaltalktypesleadstolarge variations in accuracy and may be of limited validity. For a start, we see potential toincorporatesemi-automatedtaggingusingexistingdictionariestoidentifystudents’ELlearningandcriticalthinkingskillsattheindividuallevel(Knight&Littleton,2015).Moretimeandscaffoldingareneededtohelpstudentsgaingreaterbenefitsfromthe interpretationof learningbehaviourbasedon LA dashboard data. In future iterations, we will attempt to have a longer period for self-assessmentandreflectiononWiREAD.Theteacherdashboardisalsobeingdevelopedandrefinedtoimprove the adaptability of pedagogical strategies, giving consideration to learners’ needs andinterests. The aim is to enhance the pedagogical sensitivity and responsiveness of both the LAdashboard and the teacher-pedagogue in recognition of the tight coupling between extrinsicpedagogical scaffolding and intrinsic self-regulated learning (Azevedo, Moos, Greene, Winters, &Cromley,2008).WealsoacknowledgethattheWiREADinterventioncouldbenefitfrommoreformalcurriculum timebeingdedicated to it,which is a commonstruggle facedbymany research-basedcurriculum innovation endeavors in the Singapore K–12 educational context. Having said this,WiREAD is one of the first LA-focused learning environments trialled in the Singapore secondaryschoolingcontextinvolving116studentsandthreeELteachers.Atthetimeofwriting,WiREADhasbeenadoptedbytheincubatorschoolacrossthewholeyearlevel,andtrialsarecurrentlyunderwayinthreeothersecondaryschools.

The preliminary findings reported in this paper relate only to the first of three design,implementation, and evaluation cycles currently underway. Nevertheless, it is our hope that thedata and discussion presented here fill some gaps in LA research by foregrounding secondarystudent-users’ perceptions and experiences associatedwith the design and impact of LA for theiroverall learning and growth while shedding some light on relevant aspects of Asian edu-culturalcontexts.Thispapercontributedempiricalevidenceonhowstudentsmadesenseofandaccountedfor the promise and perils of LA dashboards and visual analytics. We have also attempted tounderscoresomelessdiscernibleeducationalproblematicsthatunderlieLAdesign,relatingtobothinstrumentalandconceptualdebatesfeaturedinthefield’sscholarlydiscourse.Whiledefinitelynota silver bullet, the affordances of the LA environment tomake visible previously hidden learningbehaviours,content,andinteractionsprovideapossiblemeanstomotivateandengagelearnersincollaborativecriticalreadingasagenerativesocialpracticethatconstitutesafundamentalelementof productive 21C literacies. At the same time, the findings reported in this paper remind LAdesigners tobeawareof the restrictiveeffectsofone-sized-fits-all approaches toassessmentand



pedagogy in conventional schooling (Gašević,Dawson, Rogers,&Gašević, 2016), and the need todevelopricher,morenuancedandproximalmulti-dimensionalanalytics(Dawson&Siemens,2014).In this way, differentiated instruction can become an experienced reality for students, withpurposefully designed LA serving to compress, rather than exacerbate, the learning andachievementgapbetweenthrivingandstrugglingstudentssuchthatmoremeaningfulandequitableeducationalexperiencesandoutcomesmayberealized.

ACKNOWLEDGEMENTS

Thispaperdraws fromprojectNRF2013-EDU001-EL019, fundedby theeduLabResearchProgram,SingaporeNationalResearchFoundation.Theviewsexpressedinthispaperaretheauthors’anddonotnecessarilyrepresenttheviewsofNIESingapore.

REFERENCES

Arnold,K.E.,&Pistilli,M.D. (2012).CoursesignalsatPurdue:Using learninganalytics to increasestudent success. Proceedings of the 2nd International Conference on Learning Analytics andKnowledge (LAK ʼ12), 29 April–2May 2012, Vancouver, BC, Canada (pp. 267–270). New York:ACM.http://dx.doi.org/10.1145/2330601.2330666

Azevedo, R.,Moos, D. C., Greene, J. A.,Winters, F. I., & Cromley, J. G. (2008).Why is externallyfacilitated regulated learning more effective than self-regulated learning with hypermedia?Educational Technology Research and Development, 56(1), 45–72.http://dx.doi.org/10.1007/s11423-007-9067-0

Beaudoin,L.P.,&Winne,P.(2009).nStudy:AnInternettooltosupportlearning,collaborationandresearching learning strategies. Paper presented at the Canadian e-Learning Conference, June2009,Vancouver,BC.http://www.sfu.ca/~lpb/tr/2009-Luc_P_Beaudoin-Phil_Winne-nStudy.pdf

Beheshitha,S.S.,Hatala,M.,Gašević,D.,&Joksimović,S.(2016,April).Theroleofachievementgoalorientations when studying effect of learning analytics visualizations. Proceedings of the 6thInternational Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016,Edinburgh,UK(pp.54–63).NewYork:ACM.http://dx.doi.org/10.1145/2883851.2883904

Buckingham Shum, S., & Ferguson, R. (2012). Social learning analytics. Educational Technology &Society,15(3),3–26.

Burnett, C.,Davies, J.,Merchant,G.,&Rowsell, J. (Eds.). (2014).New literacies around the globe:Policyandpedagogy.NewYork:Routledge.

Chan,J.C.,&Lam,S.F.(2010).Effectsofdifferentevaluativefeedbackonstudents’self-efficacyinlearning.InstructionalScience,38(1),37–58.

Christakis,N.A.,&Fowler,J.H.(2009).Connected:Thesurprisingpowerofoursocialnetworksandhowtheyshapeourlives.Boston,MA:Little,Brown.

Cope,B.,&Kalantzis,M.(2015).Apedagogyofmultiliteracies:Learningbydesign.Basingstoke,UK:PalgraveMacmillan.

Corrin,L.,&deBarba,P.(2014).Exploringstudents’interpretationoffeedbackdeliveredthroughlearninganalyticsdashboards.Proceedingsofthe31stAnnualConferenceoftheAustralasianSocietyforComputersinLearninginTertiaryEducation(ASCILITE2014),23–26November2014,Dunedin,NewZealand.AustralasianSocietyforComputersinLearninginTertiaryEducation.



http://ascilite2014.otago.ac.nz/files/concisepapers/223-Corrin.pdfDavison,C.(2013).Innovationinassessment:Commonmisconceptionsandproblems.InK.Hyland&

L.L.C.Wong(Ed.),InnovationandchangeinEnglishlanguageeducation(pp.263–275).London:Routledge.

Dawson,S.,&Siemens,G.(2014).Analyticstoliteracies:Thedevelopmentofalearninganalyticsframeworkformultiliteraciesassessment.TheInternationalReviewofResearchinOpenandDistributedLearning,15(4).http://dx.doi.org/10.19173/irrodl.v15i4.1878

Deci,E.L.,&Ryan,R.M.(2011).Self-determinationtheory.Handbookoftheoriesofsocialpsychology,1,416–433.

Dweck,C.(2012).Mindset:Howyoucanfulfilyourpotential.HachetteUK.Eryilmaz,E.,vanderPol,J.,Ryan,T.,Clark,P.M.,&Mary,J.(2013).Enhancingstudentknowledge

acquisitionfromonlinelearningconversations.InternationalJournalofComputer-SupportedCollaborativeLearning,8(1),113–144.http://dx.doi.org/10.1007/s11412-012-9163-y

Ferguson,R.,&BuckinghamShum,S.(2012).Sociallearninganalytics:Fiveapproaches.Proceedingsofthe2ndInternationalConferenceonLearningAnalyticsandKnowledge(LAKʼ12),29April–2May2012,Vancouver,BC,Canada(pp.23–33).NewYork:ACM.http://dx.doi.org/10.1145/2330601.2330616

Garcia,A.,Mirra,N.,Morrell,E.,Martinez,A.,&Scorza,D.A.(2015).Thecouncilofyouthresearch:Criticalliteracyandcivicagencyinthedigitalage.Reading&WritingQuarterly,31(2),151–167.

Gašević,D.,Dawson,S.,Rogers,T.,&Gašević,D.(2016).Learninganalyticsshouldnotpromoteonesizefitsall:Theeffectsofinstructionalconditionsinpredictingacademicsuccess.TheInternetandHigherEducation,28,68–84.http://dx.doi.org/10.1016/j.iheduc.2015.10.002

Gee,J.(1991).Socio-culturalapproachestoliteracy(literacies).Annualreviewofappliedlinguistics,12,31–48.https://doi.org/10.1017/S0267190500002130

Gee,J.P.(2008).Game-likelearning:Anexampleofsituatedlearningandimplicationsforopportunitytolearn.InAssessment,equity,andopportunitytolearn(pp.200-221).CambridgeUniversityPress.doi:10.1017/CBO9780511802157.009

Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012, May). The student activity meter forawareness and self-reflection. CHI Conference on Human Factors in Computing Systems:ExtendedAbstracts(CHIEA’12),5–10May2012,Austin,TX,USA(pp.869–884).NewYork:ACM.http://dx.doi.org/10.1145/2212776.2212860

Jewitt,C.,&Kress,G.R.(Eds.).(2003).Multimodalliteracy.NewYork:Lang.Kluger,A.N.,&DeNisi,A.(1996).Theeffectsoffeedbackinterventionsonperformance:Ahistorical

review,ameta-analysis,andapreliminaryfeedbackinterventiontheory.PsychologicalBulletin,119(2),254–284.http://dx.doi.org/10.1037/0033-2909.119.2.254

Knight,S.,&Littleton,K.(2015).Discourse-centriclearninganalytics:Mappingtheterrain.JournalofLearningAnalytics,2(1),185–209.

Levin,H.M.(2012).Morethanjusttestscores.Prospects,42(3),269–284.

Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learninganalytics with learning design. American Behavioral Scientist, 57(10).doi:10.1177/0002764213479367

Martin,A.J.,&Elliot,A.J.(2015).Theroleofpersonalbest(PB)anddichotomousachievementgoals



in students’ academic motivation and engagement: A longitudinal investigation. EducationalPsychology,36(7),1–18.http://dx.doi.org/10.1080/01443410.2015.1093606

McColskey,W.,& Leary,M. R. (1985).Differential effects of norm-referenced and self-referencedfeedbackonperformanceexpectancies,attribution,andmotivation.ContemporaryEducationalPsychology,10,275–284.http://dx.doi.org/10.1016/0361-476X(85)90024-4

Miles,M.B.,&Huberman,A.M.(1994).Qualitativedataanalysis:Anexpandedsourcebook.Sage.New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard

EducationalReview,66(1),60–93.Oblinger,D.G.,&Oblinger, J. L. (2005). Is it ageor IT: First steps towardsunderstanding theNet

Generation.InD.G.Oblinger&J.L.Oblinger(Eds.),EducatingtheNetGeneration.EDUCAUSE.Ochoa,X.,Suthers,D.,Verbert,K.,&Duval,E.(2014).Analysisandreflectionsonthethirdlearning

analytics and knowledge conference (LAK 2013). Journal of Learning Analytics, 1(2), 5–22.http://dx.doi.org/10.18608/jla.2014.12.2

Paul,R.,&Elder, L. (2001).Critical thinking:Tools for takingchargeofyour learningandyour life.UpperSaddleRiver,NJ:PrenticeHall.

Rosé,C.P.,&Ferschke,O.(2016).Technologysupportfordiscussionbasedlearning:Fromcomputersupportedcollaborativelearningtothefutureofmassiveopenonlinecourses.InternationalJournalofArtificialIntelligenceinEducation,26(2),660–678.http://dx.doi.org/10.1007/s40593-016-0107-y

Rosé,C.,Wang,Y.C.,Cui,Y.,Arguello,J.,Stegmann,K.,Weinberger,A.,&Fischer,F.(2008).Analyzingcollaborativelearningprocessesautomatically:Exploitingtheadvancesofcomputationallinguisticsincomputer-supportedcollaborativelearning.InternationalJournalofComputer-SupportedCollaborativeLearning,3(3),237–271.http://dx.doi.org/10.1007/s11412-007-9034-0

Schwendimann,B.A.,Rodríguez-Triana,M.J.,Vozniuk,A.,Prieto,L.P.,Boroujeni,M.S.,Holzer,A.,...&Dillenbourg,P.(2016,April).Understandinglearningataglance:Anoverviewoflearningdashboardstudies.Proceedingsofthe6thInternationalConferenceonLearningAnalyticsandKnowledge(LAKʼ16),25–29April2016,Edinburgh,UK(pp.532–533).NewYork:ACM.https://dx.doi.org/10.1145/2883851.2883930

Shute,V.J.(2008).Focusonformativefeedback.ReviewofEducationalResearch,78(1),153–189.https://dx.doi.org/10.3102/0034654307313795

Stiggins,R.(2005).Fromformativeassessmenttoassessmentforlearning:Apathtosuccessinstandards-basedschools.ThePhiDeltaKappan,87(4),324–328.

Street,B.(2003).What’s“new”innewliteracystudies?Criticalapproachestoliteracyintheoryandpractice.CurrentIssuesinComparativeEducation,5(2),77–91.

Street,B.V.(2014).Socialliteracies:Criticalapproachestoliteracyindevelopment,ethnographyandeducation.London:Routledge.

Tan,J.P-L.(2009).Digitalkids,analoguestudents:Amixedmethodsstudyofstudents’engagementwithaschool-basedWeb2.0learninginnovation.Doctoralthesis.IDCode:30396,QueenslandUniversityofTechnology,Australia.

Tan,J.P-L.,Caleon,I.S.,Jonathan,C.R.,&Koh,E.(2014).Adialogicframeworkforassessingcollectivecreativityincomputer-supportedcollaborativeproblem-solvingtasks.ResearchandPracticeinTechnologyEnhancedLearning,9(3),411–437.



Tan,J.P-L.,&McWilliam,E.(2009).Fromliteracytomultiliteracies:Diverselearnersandpedagogicalpractice.Pedagogies:AnInternationalJournal,4(3),213–225.http://dx.doi.org/10.1080/15544800903076119

Tan,J.P-L.,&Nie,Y.Y.(2015).Theroleofauthentictasksinpromoting21stcenturylearningdispositionsinMathematics.InY.H.Cho,I.S.Caleon,&M.Kapur(Eds.),AuthenticProblemSolvingandLearninginthe21stCentury(pp.19–39).Singapore:SpringerScience+BusinessMedia.http://dx.doi.org/10.1007/978-981-287-521-1_2

Tempelaar,D.T.,Rienties,B.,&Giesbers,B.(2015).Insearchforthemostinformativedataforfeedbackgeneration:LearningAnalyticsinadata-richcontext.ComputersinHumanBehavior,47,157–167.http://dx.doi.org/10.1016/j.chb.2014.05.038

Turvey,K.(2006).Towardsdeeperlearningthroughcreativitywithinonlinecommunitiesinprimaryeducation.ComputersandEducation,46(3),309–321.http://dx.doi.org/10.1016/j.compedu.2005.11.004

Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboardapplications. American Behavioral Scientist, 57(10).http://dx.doi.org/10.1177/0002764213479363Walsh,M.(2010).Multimodalliteracy:Whatdoesitmeanforclassroompractice?AustralianJournalofLanguageandLiteracy,33(3),211.

Wang,M. T., & Eccles, J. S. (2012). Adolescent behavioral, emotional, and cognitive engagementtrajectoriesinschoolandtheirdifferentialrelationstoeducationalsuccess.JournalofResearchonAdolescence,22(1),31–39.http://dx.doi.org/10.1111/j.1532-7795.2011.00753.x

Wiliam,D.(2011).Whatisassessmentforlearning?StudiesinEducationalEvaluation,37(1),3–14.Wise,A. F. (2014,March).Designingpedagogical interventions to support studentuseof learning

analytics.Proceedingsofthe4th InternationalConferenceonLearningAnalyticsandKnowledge(LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 203–211). New York: ACM.https://dx.doi.org/10.1145/2567574.2567588

Wise,A. F.,Vytasek, J.M.,Hausknecht, S.,&Zhao,Y. (2016).Developing learninganalyticsdesignknowledge in the “middle space”: The student tuningmodel and align design framework forlearninganalyticsuse.OnlineLearning,20(2).

Wise,A.F.,Zhao,Y.,&Hausknecht,S.N.(2014).Learninganalyticsforonlinediscussions:Embeddedand extracted approaches. Journal of Learning Analytics, 1(2), 48–71.http://dx.doi.org/10.18608/jla.2014.12.4

Yin,R.K.(2009).Casestudyresearch:Designandmethods,4thed.ThousandOaks,CA:SAGE.



APPENDIX A Examples of items for each dimension of “Productive 21C Learning Dispositions” (Visualized inWiREAD student LA dashboard component “My Learning Attitudes and 21C SkillsProfile”)Dimension No.of

itemsCronbach’s

αExampleofItem(LikertScaleof1=Stronglydisagreeto7=StronglyAgree)

Criticalthinking 3 .86 WhenaviewpointispresentedinELclass,Ianalyzethesupportingevidencetoseeifitiscredible.

Creativity 4 .88 Igiveimaginative(ornovel)answersinELdiscussions/tasks.

Curiosity 8 .92 IliketoaskquestionswhenlearninginELlessons.Collaboration 5 .90 ItrymybesttocontributetogroupworkinELclasses.Open-mindedness 7 .79 Iamgenerallycautiousaboutacceptingnewideas.*

ELself-efficacy 5 .88 IamsureIcanlearntheskillstaughtinELsubjectwell.ELtask-value 5 .84 WhatIlearninELisuseful.ELengagement:behavioural 4 .85 IworkashardasIcantolearnEL.ELengagement:emotional 4 .86 IfeelinterestedwhenweworkonELtasks.ELengagement:cognitive 4 .85 Imakeupmyownexamplestohelpmeunderstand

importantconceptsinEL.Student–teacher(S–T)relatedness:communication

4 .86 ItellmyELteacheraboutmyachievementsandfailures.

S–Trelatedness:trust 5 .85 MyELteacherbelievesinmycapabilities.

S–Trelatedness:alienation 3 .71 IfeelangrywithmyELteacher.S–Trelatedness:autonomysupport

4 .85 MyELteacherencouragesmetoaskquestions.

S–Trelatedness:competencesupport

4 .85 MyELteachergivesmefeedbackthathelpsmeimprovemywork.

Masterygoals 8 .90 TheopportunitytodochallengingworkinELclassisimportanttome.

Performancegoals 6 .83 IliketobefairlyconfidentthatIcansuccessfullyperformanELtaskbeforeItryit.

Surfacelearning 10 .88 IlimitmystudytowhatisneededfortheELexamsasIthinkthereisnoneedtodoanythingextra.

Deeplearning 10 .90 IfindnewELtopicsinterestingandspendextratimetryingtolearnmoreaboutthem.

*reverse-codeditem

title learner dashboards a double-edged sword? students ... · 1.1 la affordances in k–12...

Documents